Sonification is a sensory augmentation strategy whereby a sound is associated with, and modulated by, movement. Evidence suggests that sonification could be a viable strategy to maximize learning and rehabilitation. Recent studies investigated sonification of action observation, reporting beneficial effects, especially in Parkinson’s disease. However, research on simulation training—a training regime based on action observation and motor imagery, in which actions are internally simulated, without physical execution—suggest that action observation alone is suboptimal, compared to the combined use of action observation and motor imagery. In this study, we explored the effects of sonified action observation and motor imagery on corticospinal excitability, as well as to evaluate the extent of practice-dependent plasticity induced by this training. Nineteen participants were recruited to complete a practice session based on combined and congruent action observation and motor imagery (AOMI) and physical imitation of the same action. Prior to the beginning, participants were randomly assigned to one of two groups, one group (nine participants) completed the practice block with sonified AOMI, while the other group (ten participants) completed the practice without extrinsic auditory information and served as control group. To investigate practice-induced plasticity, participants completed two auditory paired associative stimulation (aPAS) protocols, one completed after the practice block, and another one completed alone, without additional interventions, at least 7 days before the practice. After the practice block, both groups significantly increased their corticospinal excitability, but sonification did not exert additional benefits, compared to non-sonified conditions. In addition, aPAS significantly increased corticospinal excitability when completed alone, but when it was primed by a practice block, no modulatory effects on corticospinal excitability were found. It is possible that sonification of combined action observation and motor imagery may not be a useful strategy to improve corticospinal, but further studies are needed to explore its relationship with performance improvements. We also confirm the neuromodulatory effect of aPAS, but its interaction with audiomotor practice remain unclear.
Motor skill learning is a fundamental aspect of everyday life and injury recovery. Recent evidence exploring strategies to maximize learning suggest that action simulation training based on action observation (AO) and motor imagery (MI) could be an effective addition to the learning process (Ruffino et al. 2017; Ste-Marie et al. 2012), and rehabilitation (for a review see Abbruzzese et al. 2015; Mulder 2007). Neurophysiological and neuroimaging studies on AO and MI report activation of a similar fronto-parietal network to the one active during the physical execution of an action (PE; Filimon et al. 2007; Hardwick et al. 2018; Simos et al. 2017). In addition, investigations on the specificity of this neural activation using transcranial magnetic stimulation (TMS) report that during AO and MI, corticospinal excitability exhibits spatial, temporal and contextual similarities to PE (Grospretre et al. 2016; Naish et al. 2014). Recent investigations explored the combined effects of performing MI during AO, suggesting that combined action observation and motor imagery (AOMI) engages the brain in a more extended way, compared to the two activities alone. Neuroimaging studies report that, although AOMI engages largely overlapping networks also involved in AO and MI alone, it also exhibits distinctive neural signatures (Taube et al. 2015). Furthermore, studies using electroencephalography (EEG) showed that this increased activity is focused in alpha and beta frequency bands, which have also been involved in sensorimotor computations (Berends et al. 2013; Eaves et al. 2016a, b). The extended brain activity during AOMI is also reflected in an increased corticospinal excitability (Mouthon et al. 2015; Sakamoto et al. 2009; Wright et al. 2018). Evidence also shows that AOMI may enhance learning and performance compared with AO or MI alone, in sport contexts (Aoyama et al. 2020; Romano-Smith et al. 2018), rehabilitation, such as stroke (Sun et al. 2016) and Developmental Coordination Disorder (DCD; Marshall et al. 2020; Scott et al. 2019). The beneficial effect of combining AO and MI may be due to the complementarity of AO and MI. Indeed, AO engages visual as well as motor areas, to map the observed action into the observer’s own motor system (Friston et al. 2011), while MI entails similar processes involved in physical execution of the action (Hardwick et al. 2018), including its sensory afferences (Kilteni et al. 2018). To account for this, in recent years, a ‘dual simulation hypothesis’ has been proposed, which suggests that sensorimotor representations evoked by AO and MI may be computed simultaneously and, according to their content and perspective, could either complement or compete for neural substrates underlying sensorimotor computation (Eaves et al. 2016b; Vogt et al. 2013.
In the present study, we aimed at investigating whether it is possible to maximise the effect of AOMI on corticospinal activation with the use of sonification, an auditory augmentation strategy whereby an extrinsic sound is associated to—and modulated by—movement (Schaffert et al. 2019). The rationale for the combined use of sonification in conjunction with AOMI is that multisensory interaction improves perception and integration of sensory information into internal models of the body and the environment (D’Alonzo et al. 2020; Friston et al. 2010; Shams and Seitz 2008), thus enabling and improving the ability of the former to interact with the latter (Wolpert and Flanagan 2016). Sonification was suggested to be a viable strategy to improve performance and maximise learning (Dyer et al. 2015). Recent works on sonification explored its use in observational learning. Schmitz et al. (2013) asked participants to observe videos of a human-like avatar performing a breaststroke, where the relative distance between the wrists and ankles were associated with synthetized sounds. When the sound was congruent with the observed action, sonified action observation (sAO) yielded better perceptual judgment about movement speed, compared to incongruent conditions, where the matching between sound and action was not coherent and synchronised. In addition, congruent sonification activated areas known for their involvement in multisensory integration, such as the superior temporal sulcus and the insula, and it enhanced the strength of their connection with sensorimotor areas. More recently, Mezzarobba et al. (2018) reported that sAO of different actions, followed by physical execution of the same actions, significantly reduced freezing of gaze in people with Parkinson’s disease. So far, evidence is in favour of an additive learning effect of MI on AO, and of sonification on AO. However, it is currently unknown whether motor imagery could have an incremental effect on sAO. Recent evidence suggests that during MI, the brain simulates also the sensory consequences of the imagined movement (Kilteni et al. 2018), and a copy of the motor command (efference copy) is treated as a sensory afference and integrated with others sensory modalities (Pinardi et al. 2020). Thus, it is conceivable that the spatiotemporal information about an action obtained during AO and sonification, along with the simulated one during MI, would all converge to a better integration of a multisensory internal models. This would be in line with the dual simulation hypothesis, suggesting that congruent sensorimotor representations would facilitate the simulation of the action, and potentially afford plasticity (Eaves et al. 2016b). Thus, the first aim of this study was to investigate whether sonified AOMI (sAOMI) of a right-hand battery pinching would enhance motor cortex excitability. To investigate it, we compared practice-related changes in peak-to-peak amplitude of the motor-evoked potentials (MEPs) in two groups of participants undergoing a practice block based on AOMI. For one group, AOMI was enriched with sonification (SON group) and the other without extrinsic auditory information (CON group).
A secondary aim of this study was to gain information about audiomotor plasticity arising from the interaction between sonification, action observation and motor imagery (sAOMI) practice. To do so, we took advantage of the inter-dependency between practice and neuroplasticity, i.e., the propensity of the nervous system to change its structure and function with experience (Di Pino et al. 2014b). Studies on neural mechanisms of practice-dependent plasticity report that motor skill learning is associated with a long-term potentiation (LTP) of synaptic strength and weight within the network targeted by the learning process (Dayan and Cohen 2011; Rioult-Pedotti et al. 1998; Ziemann et al. 2001). The interaction of training and LTP-like plasticity can also be studied in humans using non-invasive brain stimulation (Cirillo et al. 2016). One such protocol is Paired Associative Stimulation (PAS) which is based on a repeated association between a sensory stimulus and a TMS pulse, inducing a long-lasting increase in corticospinal excitability (Carson and Kennedy 2013; Stefan et al. 2000; Suppa et al. 2017). This protocol can be harnessed to investigate the interaction between practice and plasticity. Evidence shows that practice interferes with the induction of the long-lasting increase in corticospinal excitability induced by PAS when done alone; That is, when practice precedes PAS, the induction of LTP-like plasticity is occluded (Di Pino et al. 2014b; Rosenkranz et al. 2007a, b; Stefan et al. 2006; Ziemann et al. 2004). Interestingly, recent evidence suggests that plasticity in the audiomotor pathway can be studied using PAS. Sowman et al. (2014) reported that pairing the utterance of a word 'Hey' with TMS stimulation delivered 100 ms after the sound onset at the FDI muscle increased corticospinal excitability by 40% immediately after the auditory PAS (aPAS) and 65% increase 15 min from its end. In this study, we employed aPAS to study the temporal interaction between a sAOMI practice and LTP-like plasticity of the audiomotor pathway artificially induced by a non-invasive neuromodulatory protocol. To do so, we administered aPAS to our participants after a practice session based on AOMI and compared the induced changes of motor cortex excitability with the ones induced in the same subject by aPAS alone performed on a different day.
Twenty-two self-reported neurologically and psychiatrically healthy right-handed young adults (Table 1; eight females; age: M 25.67, SE 2.08) were recruited for this study. None of them reported completing any formal musical training. Participants completed the Edinburgh Handedness Inventory to assess their hand dominance (EHI; Oldfield 1971). In addition, participants completed a TMS safety screening questionnaire (Rossi et al. 2009, 2011). Finally, participants’ vividness of MI was assessed using the Motor Imagery Questionnaire 3 (MIQ-3; Williams et al. 2012). Two participants dropped out after the first session. In addition, one more participant’s data were discarded due to compromised M-wave recording. Those participants were excluded, leaving nineteen participants to be included in the analysis. Nine participants were assigned to the SON group, and the remaining ten were assigned to the CON group. The study was approved by the Brunel University London College of Health, Medicine and Life Sciences Research Ethics Committee and data collection was in accordance with the principles of the Declaration of Helsinki.
Figure 1a provides a chronological representation of the experimental design. The experiment consisted of two sessions, completed in fixed order on 2 separate days. The second session was completed after at least 7 days, to prevent carryover influences of the aPAS on the first session (Ziemann et al. 2004). In the first session, participants completed an aPAS protocol alone. This session served as a baseline for comparison with data from the second session. Corticospinal excitability was assessed before (PRE) and after (POST) the aPAS protocol. The second session was designed to assess the influence of sonification on corticospinal excitability, and audiomotor plasticity arising from the training. Participants completed a practice block composed of congruent AOMI followed by either MI or PE of the same action (see later for more details). In this practice session, participants were randomly assigned to two groups: SON group engaged in sAOMI, while CON completed the session without extrinsic auditory information. After the practice, participants completed another aPAS protocol, which allowed us to investigate the audiomotor-induced plasticity arising from the training. In the second session, corticospinal excitability was measured at three time points: Before (PRE) and after (POST 1) the practice block, and after the aPAS session (POST 2).
Combined action observation and motor imagery practice
During the second experimental session, both groups completed a single AOMI practice block, comprising 96 trials for a total duration of approximately 30 min. Trials were split into six blocks, with a 1-minute break in between the blocks, to allow the participant to relax. Figure 1b depicts a schematic representation of the stimuli presentation during each trial. Participants first observed the action and were asked to concurrently imagine executing the same action using kinesthetic motor imagery. During AOMI, SON group listened to the sonification sound, while CON group did not hear any extrinsic sound. After that, a blue cross appeared for 1 s, notifying them to get ready, after which an icon indicated that they should either imagine (thought bubble icon) or imitate (battery icon) the action. After this, participants pushed the ‘enter’ button on a numeric keypad, to terminate the trial, and rest for 5 s, after which a new trial begun, by showing another blue cross. Ten trials in each block required the participants to engage in MI; the remaining ones required them to perform the action (PE). We chose to add physical execution trials, because we sought to design a practice protocol that was as similar as possible to one that would be carried out in applied settings. It has been argued that, although it is possible to learn an action using just MI (Kraeutner et al. 2015), physical execution of an action remains a fundamental component in motor learning (Mulder et al. 2004). Previous research has highlighted the benefits of execution trials in mental practice (Ruffino et al. 2017), and evidence from clinical studies show that people that who cannot execute movements, such as in spinal cord injury, can attempt at perform it, with beneficial effects for performance (Mateo et al. 2015) MI and PE trials were fully randomized; PE occurring 25% of trials.
Task and sonification process
Participants observed an actor pinching a battery with his right thumb and index finger, an action that was either sonified (SON) or not (CON). Sonification was performed using a frame-by-frame strategy. Raw videos were recorded using a Sony HDR-TD30, and images were acquired at 25 frames per second, at a resolution of 1920 × 1080 pixels. The raw files were exported in the free video editing software Hitfilm express 2017 (FXHOME Limited, UK) for sonification. We chose to sonify the distance between the thumb and the index finger. The sonification sound chosen was a synthesized pitch, which was created in the opensource software Audacity. The sound was first created and then matched with the video in Hitfilm express 2017 (FXHOME Limited, UK). We chose a synthetized sound because we were interested in exploring the potential use of non-action sound, auditory stimuli that do not evoke audiomotor resonance per se. Research shows that these sounds can be effectively associated to the representation of an action (Ticini et al. 2011). In addition, our audiomotor mapping is the most used in sonification research, as per a recent systematic review (Dubus and Bresin 2013).
Assessment of corticospinal excitability
To measure changes in corticospinal excitability as a result of the intervention and aPAS, we investigated changes in peak-to-peak amplitude MEPs of the right first dorsal interosseous (FDI) muscle, a muscle that was involved in the action. TMS pulses were delivered at 130% of individual’s resting motor threshold (rMT). In addition, we investigated the input–output relationship of MEPs (IO curve). For this test, MEPs were collected at the intensities of 80%, 90%, 100% (rMT), 105%, 110%, 120%, 130%, 140% and 150% of rMT. A total of 90 pulses were randomly delivered, 10 per stimulation intensity. The IO curve assesses differential recruitment of different motor units with increasing stimulation intensity (Carroll et al. 2001; Devanne, Lavoie, & Capaday, 1997). Both MEP and IO curve data were collected because the latter may be necessary when the protocol implies measuring corticospinal excitability across multiple days, as is more robust to possible confounds, such as intertrial changes in coil position and orientation (Rossini et al. 2015).
Participants sat on a chair in front of a 24″ LCD monitor (model XL2430-B, BENQ) at a viewing distance of one meter. They were instructed to position their arms and elbows on the table, keeping their hands in a pronated and relaxed position. Muscle activity was monitored throughout the experiment. Participants were continuously reminded to relax as much as possible, and not to move during the stimulation periods. TMS responses were delivered using a Magstim 200 delivering monophasic pulses (Magstim Company, Whitland, UK), using a 70 mm figure-of-eight stimulation coil, oriented as to induce posterior-to-anterior current. MEPs were collected using Ag/AgCl electrodes (Kendall, Covidien, Canada) arranged in a bipolar, belly-tendon montage. To reduce skin resistance, participants’ skin area was shaved (if necessary), abraded using an abrasive paste and cleaned using isopropyl alcohol swabs. After the preparation of the participant, the hotspot for TMS stimulation was found. Hotspot identification began by placing the coil 5 cm lateral and 1 cm anterior to the individually defined apex. From this position, the hotspot was defined as the coil position and orientation that evoked MEPs of the largest amplitude at the same stimulation intensity. The position was marked on the scalp with a soft-tip pen, to allow repositioning of the TMS coil after the breaks. Subsequently, the resting motor threshold (rMT) was determined using adaptive threshold hunting technique (Ah Sen et al. 2017; Awiszus 2011). This allowed us to determine the rMT with a reduced number of TMS pulses, thereby improving participants’ comfort, and reducing total testing time. During all periods of TMS stimulation, participants were asked to direct their visual attention to a fixation cross at the center of a screen and to count down from 200 to 0 (Kumpulainen et al. 2014). At the end of each session, FDI M-waves were collected to normalize MEPs across participants. This was done using peripheral magnetic stimulation of the ulnar nerve, which was obtained by placing the TMS coil on the elbow, between the olecranon and the medial epicondyle, with the coil handle perpendicular to the direction of the ulnar nerve, to induce current flow in the nerve with the monophasic stimulator (Lampropoulou et al. 2012). To determine M-max, we collected five evoked M-waves responses from intensities ranging from 20 to 70% of the maximum stimulator output, with incremental steps of 10%. Surface electromyography and evoked responses were recorded using Signal (v. 6, CED, UK) and amplified at a gain of 1000 and sampled at 4 kHz. To reduce the influence of external artefacts, an online band-pass filter (5–2000 Hz) was applied. TMS was applied through synchronized stimulus presentation, using TTL output triggers generated by E-Prime software (v 3.0; Psychology Software Tools, Pittsburgh, PA), and sent to the magnetic stimulator.
Auditory paired-associative stimulation (aPAS)
The aPAS protocol (Fig. 1c) consisted of 200 audiomotor pairings, each of which consisted of an auditory stimulus and a TMS pulse. The protocol was controlled using E-Prime, which was used to time the TMS pulse in relation to the auditory stimulus. The pairing auditory stimulus was a pre-recorded sound of fingers typing on a computer keyboard, and the TMS pulse was delivered 100 ms after the sound onset, with stimulus intensity set at 120% rMT. The auditory stimulus was played for 3000 ms. The interstimulus interval (ISI) between sound onset and TMS pulse was chosen in accordance with previous research on aPAS (Sowman et al. 2014). The pairs of stimuli were delivered with a random interval between 4000 and 6000 ms. The pairings were organized in 4 blocks of 50 pairings each, with one minute of rest between blocks. Auditory stimulation was delivered via in-ear earphones. Sound volume was adjusted for each participant so that it was comfortable to hear the sound, without perceived distortions. During the protocol, participants were asked to direct their gaze to a white fixation cross on the screen, and to pay attention to the sound. Prior to the beginning of the protocol, the sound was played, and all participants successfully reported to recognize the action sound.
Data and statistical analysis
MEPs Analysis All data were stored on an external drive for offline analysis. For each trial, MEPs peak-to-peak amplitude and background EMG levels were calculated using a custom-made script in Signal software (CED, v 6.05; UK), and then exported to Microsoft Excel for further analysis. Muscle activity prior to the TMS pulse was calculated as a root mean square of background EMG during the 100 ms prior to the TMS pulse. Trials with background EMG levels greater than 300 µV were excluded from MEPs analysis. With this threshold, less than 1% of the total number of MEPs were removed from the analysis. Raw MEPs were normalized and expressed as a percentage of the maximal evoked muscle response (Mmax), obtained for each participant at the end of each testing session, using the following formula (henceforth, MEPs will refer to normalized, not raw, MEPs):
We chose this normalization method because Mmax is thought to thought to be stable across time, as it represents the maximal activation of the α motoneuron pool, in this case evoked by peripheral magnetic stimulation (Lampropoulou et al. 2012; Palmieri 2004). Thus, this gave a stable comparison for MEPs, which are influenced by different activities (Bestmann et al. 2015; Klein-flu et al. 2013).
IO Curve Analysis
The relationship between TMS stimulation and MEP response, was investigated by fitting a four parameter Boltzmann sigmoid function over the MEPs of the nine stimulation intensities. Peak-to-peak amplitude and bgEMG was calculated using the same script. We averaged MEPs for each stimulation intensity. Curve fitting was performed using the built-in sigmoid curve fitting features of Signal software (CED, v 6.05, UK). The fitting was done using the following equation:
where MEPmax and MEPmin are the maximum and minimum asymptote, respectively; I50 is the stimulus intensity needed to evoke MEPs that are 50% of MEPmax, and s is the slope of the curve. Curve fitting with Boltzmann equation provided several parameters, which were then used to characterize changes in corticospinal excitability as a result of protocol intervention (Carroll et al. 2001; Devanne et al. 1997). In addition to the parameter in the equation above, another index was calculated, slope I50, which represented the slope of the ascending phase of the curve at I50, which was calculated according to the following formula:
where m is the slope parameter of the Boltzmann sigmoid function.
Statistical Analysis Statistical analysis was carried out in SPSS. Outliers in the data were assessed using z scores. Values greater than ± 2.99 were considered outliers and discarded from the analysis. Data distribution was assessed via the Shapiro–Wilk test. A paired-sample t test was used to assess statistical differences in rMT between sessions, while an independent t test was used to assess group difference in rMT in the second visit. The same tests were also used to assess between groups differences in vividness of motor imagery, by analyzing the three output of the MIQ questionnaire, internal visual imagery (IVI), external visual imagery (EVI) and kinesthetic motor imagery (KI). Lastly, between-days changes in Mmax were calculated using a paired-sample t-test. Some of the indices were not normally distributed (p > 0.05), so non-parametric statistical analyses were used instead. Homogeneity of variance was assessed using Levene's test for equality of variances. To assess the effects of aPAS alone (on experimental session 1), we performed non parametric test on pre- and post-aPAS MEPs. Wilcoxon signed-rank test was used to assess statistical differences on IO curve indices. To assess the effects of the practice block on corticospinal excitability, and its priming effect for aPAS, we performed a mixed ANOVA with factors TIME and GROUP. TIME factor had three levels—pre-training, post-training (post 1) and post-aPAS (post 2), and GROUP two (SON and CON). In addition, we also analyzed the percentage change of corticospinal excitability after the two sessions. To this end, we performed a mixed ANOVA with factors ‘TIME’ (two levels:’aPAS D1′ and’aPAS D2′) and GROUP (two levels, SON and CON). For IO curve indices, six parameters were analyzed, MEPmax, MEPmin, MEP range, slope, I50 and slope I50. For each of these indices, an individual Sphericity of covariance was assessed with Mauchly's test of sphericity. In case of violation of sphericity, Greenhouse–Geisser epsilon adjustment was used. Bonferroni correction was applied for post hoc comparisons.
There were no significant differences in rMT between the first (38 ± 5%) and the second visit (38 ± 4%), t(19) = 0.151, p = 0.882. In addition, there were no statistically significant differences in rMT between the SON (37 ± 4%) and CON group (38 ± 4%) on the second visit, t(17) = − 0.612, p = 0.55. The MIQ-3 analysis showed no significant differences between the groups in self-reported vividness of Internal Visual Imagery (t(19) = − 0.49, p = 0.63), External Visual Imagery (t(19) = 0,62, p = 0.54), or Kinaesthetic Imagery (t(19) = − 0.36, p = 0.72). No statistically significant differences were found in Mmax between the first (11.38 ± 4.22 mV) and the second (11.70 ± 4.52 mV) visits, as assessed using a Wilcoxon signed-rank test (z = − 0.181, p = 0.856).
Session 1: Effects of aPAS on corticospinal excitability
Figure 2 and Table 2 provide a summary of the results for the first session. The aPAS protocol induced a significant increase in peak-to-peak MEP size (Fig. 2a), as compared with pre-aPAS measure: z = 3.058, p = 0.002). Figure 2d reports the IO curve fitting with Boltzmann function. Analysis on the indices arising from curve fitting reported a significant increase in MEPmax (z = 2.495, p = 0.013; Fig. 2b), slope of the fitted curve (z = 2.012, p = 0.44, Fig. 2c), and range of MEP responses (z = 2.535, p = 0.11). No significant differences were found for MEPmin (p = 0.136), I50 (p = 0.390), and the slope at I50 (p = 0.601).
Session 2: Effects of AOMI training practice on corticospinal excitability and practice-dependent plasticity
Figure 3 and Table 3 provide a summary of the main results for the second session. There was a main effect of ‘TIME’ on peak-to-peak MEP amplitude: F(2,34) = 7.397, p = 0.002, η2p = 0.303. No interaction TIME x GROUP on peak-to-peak MEP amplitude was found: F(2,34) = 0.972, p = 0.389, η2p = 0.054. Post hoc analysis using Bonferroni correction revealed that MEP mean amplitude significantly increased after the training, as compared with pre-training values (p = 0.015, Fig. 3a). No significant changes were found between POST1 and POST2, suggesting that post-training aPAS did not significantly change corticospinal excitability. For the analysis of the parameters IO curve (Fig. 3b) arising from the Boltzmann fitting, a mixed ANOVA reported no main effects of TIME, nor TIME x GROUP interaction for any parameter (details on statistical analysis reported in table S1, supplementary material).
Between-days effects of aPAS
Both groups showed a decrease in aPAS effect on the second visit, compared to the first one, expressed as a post-aPAS percentage change in MEP peak-to-peak amplitude (Fig. 3c, Table 4). A mixed ANOVA revealed a main effect of TIME on percentage change of MEP peak-to-peak amplitude following the aPAS on the two experimental sessions: F(1,17) = 8.183, p = 0.011; η2p = 0.325. No interactions TIME x GROUP were found: F(1,17) = 1.275, p = 0.274; η2p = 0.07.
This study was carried out to investigate the effects of sonification of combined action observation and motor imagery on corticospinal excitability. To this purpose, we trained participants to engage in a practice block comprising congruent AOMI, MI and execution of the same action. The experimental group received sonification during AOMI, while a control group received no sonification. An additional aim of this study was to investigate audiomotor plasticity arising from such training. To do so, we used a variation of an established method to investigate neural plasticity, auditory paired-associative stimulation.
Combined action observation and motor imagery training and effect of sonification
The primary aim of this project was to investigate the effects of auditory augmentation of AOMI on corticospinal excitability. To this end, participants completed a single practice session based on AOMI, MI and physical execution of the same action. In addition, a SON group received auditory augmentation during AOMI. Sonification yielded no significant facilitation of corticospinal excitability, compared to training without sensory augmentation. Although we are not aware of studies exploring the effects of sonification of corticospinal excitability, neuroimaging and behavioral studies have shown that observing a sonified action induces better movement-related perceptual judgments, a more active engagement of the sensorimotor system during AO (Schmitz et al. 2013), as well as superior performance and rehabilitative outcomes in people with Parkinson’s disease (Mezzarobba et al. 2018). It is possible that sonification did not exert its enriching function during AOMI because the task was straightforward to perform or imagine, rendering the auditory information redundant. There is evidence suggesting that corticospinal excitability is influenced by the vividness of MI (Lebon et al. 2012). Thus, even though the task was straight forward, it may not necessarily mean that it was easy to imagine. However, MIQ results suggest that our participants were, on average, good imagers (c.f. Marchesotti et al. 2016; Vuckovic and Osuagwu 2013), thus further decreasing the value of sensory augmentation. Given the need for accurate coil localization, we were restricted on actions that could be used in this study. Future studies should explore sonification of simulated action using a more ecologically valid action accordingly.
Another possible reason for the lack of effect of sonification on corticospinal excitability may be due to interactions between AO, MI, and external auditory feedback. Recent investigations suggest that combined usage of AO and MI affects attentional processing and mental effort (Bruton et al. 2020; Meers et al. 2020). Studies show that during AOMI, there is a reallocation of attention between externally evoked to internal simulation of the kinesthetic predicted sensation arising from the action (Eaves et al. 2016a, b). Studies investigating corticospinal excitability during various forms of AOMI support this view. Bruton et al. (2020) assessed corticospinal excitability, eye movement and behavioral data while participants engaged in congruent, coordinated, and conflicting AOMI. Congruent AOMI, as used in this study, resulted in significantly higher MEPs and reduced mental effort. Relevant to the present study, however, is the fact that participants reported increased attentional demands during conflicting AOMI, and MEPs were significantly lower than during congruent AOMI. Even though research on sonification suggests that an optimal audiomotor mapping decreases attentional demands and cognitive load of the task (Dyer et al. 2017), and improves performance (Sigrist et al. 2013), there is also evidence suggesting that, compared to other sensory augmentation strategies, sonification may represent an additional attention weight on people, especially early in the training regime (Ronsse et al. 2011). In our study, we used congruent AOMI, which has been shown to require less mental effort, but the addition of sonification may have resulted in comparable increases in attentional demands, thereby negating potential facilitative effects of the former. Further studies are needed to confirm this hypothesis.
Regardless of sonification, however, statistical analysis revealed a practice effect that agrees with the available literature on practice-related neuromodulation. Thus, the training exerted its modulatory effect. Motor learning, with or without sensory augmentation, is characterized by an increase in corticospinal excitability, as measured by TMS (Jung and Ziemann 2009; Rosenkranz et al. 2007a, b; Ziemann et al. 2004). It is thought that the initial phase of learning, the within-session fast learning, is based on an unmasking of silent connections, which are based on LTP-like mechanisms (Pascual-Leone et al. 1995). Studies show that even very simple movements, such as repeated thumb abduction/adduction, produce measurable changes in corticospinal excitability, in line with LTP-like plasticity (Rosenkranz et al. 2007a, b; Ziemann et al. 2004). This mechanistic view of motor learning also applies to more cognitive forms of motor learning, such as AO and MI, as evidence shows that similar plasticity-related modulation of corticospinal excitability are obtained when PAS follows a practice session of observational or mental practice (Avanzino et al. 2015; Lepage et al. 2012). In addition, engaging in AOMI may be better than AO and MI alone (Marshall et al. 2020; Marshall et al. 2019), as it has been linked to increased neural activity (Bruton et al. 2020; Eaves et al. 2016a, b; Wright et al. 2018), thus could potentially influence the rate of practice-dependent plasticity (Eaves et al. 2016). To our knowledge no research has been done on this. Taken together, our results confirm that practicing the pinching of a small object—in this case, a battery—induces an increase in corticospinal excitability of the FDI muscle. The fact that only MEPs, but not the IO curve parameters, exhibited modulation effects suggests that any learning effect was probably small.
In this study, we focussed on sonification of congruent AOMI, which has been the most studied form of AOMI. However, future studies should also explore the effects of sonification of other types of AOMIs such as coordinative and incongruent AOMI (Eaves et al. 2016b; Vogt et al. 2013). Under the dual simulation hypothesis, when the observed and imagined action are not congruent, there is a representational conflict, which results in a lower corticospinal excitability, and an increase in attentional demand to complete the task (Bruton et al. 2020; Meers et al. 2020). However, these forms of dual representation of action can still be used in motor (re)learning and should be further explored. Considering that AOMI implies a change in focus between externally top internally driven action simulation (Eaves et al. 2016a, b; Eaves et al. 2016), sonification could be used to integrate multimodal representation of a complementary aspect of an imagined action. In a hypothetical scenario, a person could imagine performing an action, while simultaneously observing the same action from another point of view and listening to auditory augmentation. Future studies, however, need to further explore whether this hypothesis could have real application to the field of motor (re)learning.
Our discussion regarding the effectiveness, or lack thereof, of sonification for simulation training remains somewhat speculative, given the inconclusive findings. Indeed, the sample size was limited, thus affecting our ability to conclusively discuss the impact of sAOMI for action simulation. Further studies, with a larger sample size, are needed, to further explore this area. Different studies have highlighted the importance of AO and MI for rehabilitation regimes, and its fundamental role in neurological conditions (Abbruzzese et al. 2015; Marshall et al. 2020; Mulder 2007) and immobilization (Bassolino et al. 2014). Under the right conditions, sonification could represent important strategy to maximize learning in clinical conditions, such as stroke survivors (Scholz et al. 2014, 2015, 2016), but could also be a viable sensory substitution strategy for conditions such as deafferentation (Danna and Velay 2017; Danna et al. 2015). Lastly, further development of sonification research may find application in the field of brain-computer interfaces, by affording strategies to improve embodiment of non-body objects, such as neuroprostheses (D’Alonzo et al. 2019; Di Pino et al. 2014a, b, 2020), an issue that crucial for optimal development of the field (Makin et al. 2017).
The effect of aPAS on corticospinal excitability
On the first visit, we evaluated the effects of aPAS on corticospinal excitability. aPAS produced an increase in MEPs immediately post aPAS, compared to pre-aPAS measures. In addition, for the IO curve parameters resulting from the Boltzmann curve fitting, we observed a significant increase in the maximum evoked potential, as well as a significant shift to the left of the slope of the curve, which is usually interpreted as an increase in corticospinal excitability (Rosenkranz et al. 2007a, b). A significant increase in the range of the evoked potentials is also consistent with the increase in MEPmax. Our results confirm those of Sowman et al. (2014), who first reported associative LTP-like plasticity within the audiomotor domain by associating a speech sound (the word ‘Hey’) to TMS delivered over the FDI muscle. In our experiment, we used a similar protocol, except that the sound associated to the TMS pulse was a keyboard typing action sound. We used this sound because we stimulated the FDI muscle, which is a prime mover for this action. Our results, however, are very similar to those obtained by Sowman and colleagues. Thus, together with this previous study, our findings suggest that the association of an action sound, regardless of the effector, to a TMS pulse delivered 100 ms after the sound onset at 120% of the individually defined rMT yields a robust modulatory effect on corticospinal excitability.
From a mechanistic point of view, PAS is based on spike-timing-dependent plasticity (STDP). One of the key features of STDP is associativity; that is, its modulating effects are based on the timing of arrival of the two stimuli on the target neuron (Suppa et al. 2017). In most of PAS interventions, an interstimulus interval of 25 ms is usually chosen to induce LTP-like plasticity (Carson and Kennedy 2013; Ranieri et al. 2019; Stefan et al. 2000). We based our protocol on an already published literature on associative plasticity in the audiomotor domain (Sowman et al. 2014). 100 ms from a stimulus onset also coincides with the N100 component of the ERP waveform, which is thought to be related to stimulus-dependent arousal (Naatanen et al. 2011; Nash and Williams 1982). There is evidence that the auditory N100 is influenced by habituation. Indeed, Löfberg and colleagues reported that, when the same auditory stimulation is delivered in trains of four—one per second—corticospinal excitability is increased only for the first stimulus in each train; subsequent TMS pulses yield decreases in corticospinal excitability, suggesting a habituation effect (Löfberg et al. 2018,2014). Nevertheless, we did not find evidence of habituation, as our results confirm that aPAS is a robust technique for modulating corticospinal excitability, consistent with Hebbian learning. In addition, evidence from visuomotor PAS confirms a modulation of corticospinal excitability with an interstimulus interval of 100 and 120 ms (Suppa et al. 2015). Taken together this raise the possibility that the interstimulus interval for cross-modal PAS may be around 100 ms. Further studies are needed, however, to confirm this hypothesis.
Occlusion of LTP-like plasticity after training
A secondary aim of this study was to gain information on the interaction between sonification and plasticity. To this end, both SON and CON group underwent an additional aPAS protocol after the training session. Evidence suggests that PAS and practice-dependent plasticity share similar neural mechanisms, such that the priming of practice affects the modulatory effects of PAS. Specifically, studies suggest that both motor skill learning and PAS-induced associative plasticity result from a modulation of synaptic strength and weight within the network targeted by the intervention, and this is based on STDP (Caporale and Dan 2008). Evidence also shows that if two LTP-inducing protocol are done in succession, the first protocol interferes with the effect of the second. This form of metaplasticity—plasticity of plasticity—can be induced with two excitatory PAS (Müller-Dahlhaus et al. 2015; Müller et al. 2007) or by priming a PAS with a practice block (Rosenkranz et al. 2007a, b; Stefan et al. 2006; Ziemann et al. 2004).
In our study, both CON and SON performed the same protocol, except for auditory augmentation during AOMI. Considering that in the first session we confirmed the sensitivity of aPAS to audiomotor plasticity, we wanted to explore the interaction between sonification and aPAS, which is designed to test audiomotor connectivity. This could provide evidence of practice-dependent cross-modal interaction. Post-aPAS measures of corticospinal excitability, however, did not report any neuromodulation, compared with post-practice measures for both groups. In addition, for both SON and CON group, the effect of aPAS completed after the practice was lower than the one completed in the first session, and no differences between the groups were found. It is possible that the effect of sonification on learning was small, and that the execution component of the training block masked any effect of sonification. The auditory cortex and M1 do not have direct connections (Cammoun et al. 2015) and, as for the visual processing (Milner and Goodale 2008), auditory processing engages two pathways, a ventral and a dorsal one (Rauschecker and Tian 2000), with the dorsal pathway being responsible for audiomotor integration (Baumann et al. 2007). It is thought that an auditory stimulus engages the motor system via the dorsal route (Rauschecker 2011), which from the thalamus, engages the parietal cortex, where it is integrated with visual and other stimuli (Tanaka and Kirino 2018), to create a multisensory perception (Gottlieb 2007). As highlighted earlier in the text, if a practice block is followed by PAS protocol, an interaction between the two protocols is evident (Rosenkranz 2007a, b; Stefan et al. 2006; Ziemann et al. 2004). MEPs are a motor phenomenon (Hallett 2007; Terao and Ugawa 2002) and, as such, it is possible that the physical execution portion of the training produced a ceiling effect in terms of LTP in M1, so as to mask any effect of sonification on the interaction between practice and aPAS.
Another possible explanation for our aPAS results could be methodological. It is also possible that the temporal spacing between the practice block and subsequent aPAS session influenced participants’ attention levels. There is evidence that the interaction between LTP-like neuromodulatory protocols are sensitive to the spacing between those two protocols (Müller-Dahlhaus et al. 2015). While plasticity arising from motor learning is long-lasting (Dayan and Cohen 2011), the spacing between our two protocols may have affected the level of attention during aPAS. This view is supported by evidence that participants’ level and focus of attention affect the outcome of PAS (Kamke et al. 2012, 2016; Stefan et al. 2004). That is, it is possible that participants may have been unable to sustain high level of attention to the protocol, or worst may have been in a state of drowsiness. However, drowsiness is associated with a decrease in corticospinal excitability (Salih et al. 2005). The fact that after the practice block the effect of aPAS on corticospinal excitability was smaller than the session completed in isolation may be evidence of a suboptimal level of attention to the aPAS stimuli. To mitigate loss of attention, future studies should explore the optimal length of an aPAS protocol, to suggest the minimum number of audiomotor pairing that still neuromodulate corticospinal excitability. It is possible that a shorter aPAS protocol may allow participants to better sustain the practice block and the aPAS protocol. For example a longer break between practice block and post-practice aPAS may be longer, to give participants time to relax, and be more predisposed to the protocol. Lastly, future studies should investigate the neural aftereffects of aPAS to gain evidence on the interaction between the practice block and aPAS, for example with combined TMS-EEG (Hallett et al. 2017; Ilmoniemi and Kičić 2010; Rogasch and Fitzgerald 2013).
To the best of our knowledge, we are the first to explore audiomotor metaplasticity. Our results extend the findings of Sowman et al. (2014), with regard to the effects of aPAS on corticospinal excitability of hand muscles. However, we acknowledge that our study would have benefitted by a larger sample size. Future studies are needed to further explore this protocol. Plasticity is thought to be the underlying neural substrate of learning, and measuring the neuromodulation resulting from the learning process is fundamental for the development of new tools and strategies to maximize learning, and more studies are needed to further elucidate the neuromodulatory effects of aPAS. The development of aPAS may provide an effective test to assess audiomotor connectivity, which may provide, in turn, mechanistic evidence for clinical deficits, as well as the link between the deficit and interventions, via occlusion of LTP-like plasticity (Rosenkranz et al. 2007a, b; Ziemann et al. 2004). Further, aPAS may also represent an intervention tool. Recent studies highlighted the potential therapeutic benefits of using PAS in neurological conditions such as stroke (Silverstein et al. 2019) or incomplete spinal cord injury (Ling et al. 2020); along those lines, aPAS may represent a viable intervention for audiomotor conditions, such as stuttering (Sares et al. 2020). To achieve this, future studies should confirm the optimal ISI. Since PAS is based on STDP (Caporale and Dan 2008), the timing of the arrival of volleys at M1 is crucial.
In the present study, we investigated the effects of sAOMI on corticospinal excitability, and its neuromodulatory role when paired with aPAS. After a training practice based on sAOMI and physical execution of the action, corticospinal excitability was not modulated, compared to pre-practice measures. In addition, our results confirm previous evidence that aPAS alone modulates corticospinal excitability, evidenced by post-aPAS increases in MEP amplitudes. However, its effects on homeostatic metaplasticity are unclear, and future studies with a larger participant pool may provide more robust evidence of the effects of sonification on action simulation training and audiomotor metaplasticity.
Abbruzzese G, Avanzino L, Marchese R, Pelosin E (2015) Action Observation and Motor Imagery: Innovative Cognitive Tools in the Rehabilitation of Parkinson’s Disease, 2015. https://doi.org/10.1155/2015/124214
Ah Sen CB, Fassett HJ, El-Sayes J, Turco CV, Hameer MM, Nelson AJ (2017) Active and resting motor threshold are efficiently obtained with adaptive threshold hunting. PLoS ONE 12(10):1–9. https://doi.org/10.1371/journal.pone.0186007
Aoyama T, Kaneko F, Kohno Y (2020) Motor imagery combined with action observation training optimized for individual motor skills further improves motor skills close to a plateau. Hum Mov Sci 73:102683. https://doi.org/10.1016/j.humov.2020.102683
Avanzino L, Gueugneau N, Bisio A, Ruggeri P, Papaxanthis C, Bove M (2015) Motor cortical plasticity induced by motor learning through mental practice. Front Behav Neurosci 9:105. https://doi.org/10.3389/fnbeh.2015.00105
Awiszus F (2011) Fast estimation of transcranial magnetic stimulation motor threshold: Is it safe? Brain Stimul 4(1):50–57. https://doi.org/10.1016/j.brs.2010.06.002
Bassolino M, Campanella M, Bove M, Pozzo T, Fadiga L (2014) Training the motor cortex by observing the actions of others during immobilization. Cereb Cortex 24(12):3268–3276. https://doi.org/10.1093/cercor/bht190
Baumann S, Koeneke S, Schmidt CF, Meyer M, Lutz K, Jancke L (2007) A network for audio-motor coordination in skilled pianists and non-musicians. Brain Res 1161(1):65–78. https://doi.org/10.1016/j.brainres.2007.05.045
Berends HI, Wolkorte R, Ijzerman MJ, Van Putten MJAM (2013) Differential cortical activation during observation and observation-and-imagination. Exp Brain Res 229(3):337–345. https://doi.org/10.1007/s00221-013-3571-8
Bestmann S, de Berker AO, Bonaiuto J (2015) Understanding the behavioural consequences of noninvasive brain stimulation. Trends Cogn Sci 19(1):13–20. https://doi.org/10.1016/j.tics.2014.10.003
Bruton AM, Holmes PS, Eaves DL, Franklin ZC, Wright DJ (2020) Neurophysiological markers discriminate different forms of motor imagery during action observation. Cortex 124:119–136. https://doi.org/10.1016/j.cortex.2019.10.016
Cammoun L, Thiran JP, Griffa A, Meuli R, Hagmann P, Clarke S (2015) Intrahemispheric cortico-cortical connections of the human auditory cortex. Brain Struct Funct 220(6):3537–3553. https://doi.org/10.1007/s00429-014-0872-z
Caporale N, Dan Y (2008) Spike timing—dependent plasticity: a hebbian learning rule. Annu Rev Neurosci 31:25–46. https://doi.org/10.1146/annurev.neuro.31.060407.125639
Carroll TJ, Riek S, Carson RG (2001) Reliability of the input–output properties of the cortico-spinal pathway obtained from transcranial magnetic and electrical stimulation. J Neurosci Methods 112(2):193–202. https://doi.org/10.1016/S0165-0270(01)00468-X
Carson RG, Kennedy NC (2013) Modulation of human corticospinal excitability by paired associative stimulation. Front Hum Neurosci 7:823. https://doi.org/10.3389/fnhum.2013.00823
Cirillo G, Di Pino G, Capone F, Ranieri F, Florio L, Todisco V, Di Lazzaro V (2016) Neurobiological after-effects of non-invasive brain stimulation. Brain Stimul. https://doi.org/10.1016/j.brs.2016.11.009
D’Alonzo M, Mioli A, Formica D, Vollero L, Di Pino G (2019) Different level of virtualization of sight and touch produces the uncanny valley of avatar’s hand embodiment. Sci Rep 9(1):1–11. https://doi.org/10.1038/s41598-019-55478-z
D’Alonzo M, Mioli A, Formica D, Di Pino G (2020) Modulation of body representation impacts on efferent autonomic activity. J Cogn Neurosci. https://doi.org/10.1162/jocn_a_01532
Danna J, Velay J (2017) On the auditory-proprioception substitution hypothesis: movement sonification in two deafferented subjects learning to write new characters. Front Neurosci. https://doi.org/10.3389/fnins.2017.00137
Danna J, Fontaine M, Paz-Villagran V, Gondre C, Thoret E, Aramaki M, Velay JL (2015) The effect of real-time auditory feedback on learning new characters. Hum Mov Sci 43:216–228. https://doi.org/10.1016/j.humov.2014.12.002
Dayan E, Cohen LG (2011) Neuroplasticity subserving motor skill learning. Neuron 72(3):443–454. https://doi.org/10.1016/j.neuron.2011.10.008
Devanne H, Lavoie BA, Capaday C (1997) Input-output properties and gain changes in the human corticospinal pathway. Exp Brain Res 114(2):329–338. https://doi.org/10.1007/PL00005641
Di Pino G, Maravita A, Zollo L, Guglielmelli E, DiLazzaro V (2014a) Augmentation-related brain plasticity. Front Syst Neurosci. https://doi.org/10.3389/fnsys.2014.00109
Di Pino G, Pellegrino G, Capone F, DiLazzaro V (2014b) Human cerebral cortex metaplasticity and stroke recovery. Austin J Cerebrovasc Dis Stroke 1(1):1–2. https://doi.org/10.13140/2.1.2221.8886
Di Pino G, Romano D, Spaccasassi C, Mioli A, Alonzo MD, Sacchetti R, Valle G (2020) Sensory- and action-oriented embodiment of neurally-interfaced robotic hand prostheses. Front Neurosci. https://doi.org/10.3389/fnins.2020.00389
Dubus G, Bresin R (2013) A systematic review of mapping strategies for the sonification of physical quantities. PLoS ONE. https://doi.org/10.1371/journal.pone.0082491
Dyer JF, Stapleton P, Rodger WMM (2015) Sonification as concurrent augmented feedback for motor skill learning and the importance of mapping design. Open Psychol J 8(1):192–202. https://doi.org/10.2174/1874350101508010192
Dyer JF, Stapleton P, Rodger M (2017) Mapping sonification for perception and action in motor skill learning. Front Neurosci 11:1–4. https://doi.org/10.3389/fnins.2017.00463
Eaves DL, Behmer LP, Vogt S (2016a) EEG and behavioural correlates of different forms of motor imagery during action observation in rhythmical actions. Brain Cogn 106:90–103. https://doi.org/10.1016/j.bandc.2016.04.013
Eaves DL, Riach M, Holmes PS, Wright DJ (2016b) Motor imagery during action observation: a brief review of evidence, theory and future research opportunities. Front Neurosci. https://doi.org/10.3389/fnins.2016.00514
Filimon F, Nelson JD, Hagler DJ, Sereno MI (2007) Human cortical representations for reaching: Mirror neurons for execution, observation, and imagery. Neuroimage 37(4):1315–1328. https://doi.org/10.1016/j.neuroimage.2007.06.008
Friston KJ, Daunizeau J, Kilner J, Kiebel SJ (2010) Action and behavior: a free-energy formulation. Biol Cybern 102(3):227–260. https://doi.org/10.1007/s00422-010-0364-z
Friston K, Mattout J, Kilner J (2011) Action understanding and active inference. Biol Cybern 104(1–2):137–160. https://doi.org/10.1007/s00422-011-0424-z
Gottlieb J (2007) From thought to action: the parietal cortex as a bridge between perception, action, and cognition. Neuron. https://doi.org/10.1016/j.neuron.2006.12.009
Grospretre S, Ruffino C, Lebon F (2016) Motor imagery and cortico-spinal excitability: a review. Eur J Sport Sci 13(3):317–324. https://doi.org/10.1080/17461391.2015.1024756
Hallett M (2007) Transcranial magnetic stimulation: a primer. Neuron 55(2):187–199. https://doi.org/10.1016/j.neuron.2007.06.026
Hallett M, Di Iorio R, Maria Rossini P, Park JE, Chen R, Celnik P, Ugawa Y (2017) Contribution of transcranial magnetic stimulation to assessment of brain connectivity and networks. Clin Neurophysiol 484:81–85. https://doi.org/10.1016/j.clinph.2017.08.007
Hardwick RM, Caspers S, Eickhoff SB, Swinnen SP (2018) Neural correlates of action: comparing meta-analyses of imagery, observation, and execution. Neurosci Biobehav Rev. https://doi.org/10.1016/j.neubiorev.2018.08.003
Ilmoniemi RJ, Kičić D (2010) Methodology for combined TMS and EEG. Brain Topogr 22(4):233–248. https://doi.org/10.1007/s10548-009-0123-4
Jung P, Ziemann U (2009) Homeostatic and nonhomeostatic modulation of learning in human motor cortex. J Neurosci 29(17):5597–5604. https://doi.org/10.1523/JNEUROSCI.0222-09.2009
Kamke MR, Hall MG, Lye HF, Sale MV, Fenlon LR, Carroll TJ, Mattingley JB (2012) Visual attentional load influences plasticity in the human motor cortex. J Neurosci 32(20):7001–7008. https://doi.org/10.1523/JNEUROSCI.1028-12.2012
Kamke MR, Nydam AS, Sale MV, Mattingley JB (2016) Associative plasticity in the human motor cortex is enhanced by concurrently targeting separate muscle representations with excitatory and inhibitory protocols. J Neurophysiol 115(4):2191–2198. https://doi.org/10.1152/jn.00794.2015
Kilteni K, Andersson BJ, Houborg C, Ehrsson HH (2018) Motor imagery involves predicting the sensory consequences of the imagined movement. Nat Commun. https://doi.org/10.1038/s41467-018-03989-0
Klein-flu MC, Nobbs D, Pitcher JB, Bestmann S (2013) Variability of human corticospinal excitability tracks the state of action preparation. Journal of Neurosci 33(13):5564–5572. https://doi.org/10.1523/JNEUROSCI.2448-12.2013
Kraeutner SN, MacKenzie LA, Westwood DA, Boe SG (2015) Characterizing skill acquisition through motor imagery with no prior physical practice. J Exp Psychol Hum Percept Perform 42(2):257–265. https://doi.org/10.1037/xhp0000148
Kumpulainen S, Avela J, Gruber M, Bergmann J, Voigt M, Linnamo V, Mrachacz-Kersting N (2014) Differential modulation of motor cortex plasticity in skill- and endurance-trained athletes. Eur J Appl Physiol 115(5):1107–1115. https://doi.org/10.1007/s00421-014-3092-6
Lampropoulou SI, Nowicky AV, Marston L (2012) Magnetic versus electrical stimulation in the interpolation twitch technique of elbow flexors. J Sports Sci Med 11(4):709–718
Lebon F, Byblow WD, Collet C, Guillot A, Stinear CM (2012) The modulation of motor cortex excitability during motor imagery depends on imagery quality. Eur J Neurosci 35(2):323–331. https://doi.org/10.1111/j.1460-9568.2011.07938.x
Lepage J-F, Morin-Moncet O, Beaule V, de Beaumont L, Champoux F, Theoret H (2012) Occlusion of LTP-like plasticity in human primary motor cortex by action observation. PLoS ONE 7(6):3–8. https://doi.org/10.1371/journal.pone.0038754
Ling YT, Alam M, Zheng YP (2020) Spinal cord injury: lessons about neuroplasticity from paired associative stimulation. Neuroscientist 26(3):266–277. https://doi.org/10.1177/1073858419895461
Löfberg O, Julkunen P, Pääkkönen A, Karhu J (2014) The auditory-evoked arousal modulates motor cortex excitability. Neuroscience 274:403–408. https://doi.org/10.1016/j.neuroscience.2014.05.060
Löfberg O, Julkunen P, Kallioniemi E, Pääkkönen A, Karhu J (2018) Modulation of motor cortical excitability with auditory stimulation. J Neurophysiol 120(3):920–925. https://doi.org/10.1152/jn.00186.2017
Makin TR, De Vignemont F, Faisal AA (2017) Neurocognitive barriers to the embodiment of technology. Nat Biomed Eng. Nature Publishing Group. https://doi.org/10.1038/s41551-016-0014
Marchesotti S, Bassolino M, Serino A, Bleuler H, Blanke O (2016) Quantifying the role of motor imagery in brain-machine interfaces. Sci Rep 6(1):24076. https://doi.org/10.1038/srep24076
Marshall B, Wright DJ, Holmes PS, Wood G, Marshall B, Wright DJ, Wood G (2019) Combining action observation and motor imagery improves eye—hand coordination during novel visuomotor task performance combining action observation and motor imagery improves. J Mot Behav. https://doi.org/10.1080/00222895.2019.1626337
Marshall B, Wright DJ, Holmes PS, Williams J, Wood G (2020) Combined action observation and motor imagery facilitates visuomotor adaptation in children with developmental coordination disorder. Res Dev Disabil. https://doi.org/10.1016/j.ridd.2019.103570
Mateo S, Di Rienzo F, Bergeron V, Guillot A, Collet C, Rode G (2015) Motor imagery reinforces brain compensation of reach-to-grasp movement after cervical spinal cord injury. Front Behav Neurosci 9:234. https://doi.org/10.3389/fnbeh.2015.00234
Meers R, Nuttall HE, Vogt S (2020) Motor imagery alone drives corticospinal excitability during concurrent action observation and motor imagery. Cortex 126:322–333. https://doi.org/10.1016/j.cortex.2020.01.012
Mezzarobba S, Grassi M, Pellegrini L, Catalan M, Kruger B, Furlanis G, Bernardis P (2018) Action observation plus sonification. A novel therapeutic protocol for Parkinson’s patient with freezing of gait. Front Neurol. https://doi.org/10.3389/fneur.2017.00723
Milner AD, Goodale MA (2008) Two visual systems re-viewed. Neuropsychologia 46(3):774–785. https://doi.org/10.1016/j.neuropsychologia.2007.10.005
Mouthon A, Ruffieux J, Wälchli M, Keller M, Taube W (2015) Task-dependent changes of corticospinal excitability during observation and motor imagery of balance tasks. Neuroscience 303:535–543. https://doi.org/10.1016/j.neuroscience.2015.07.031
Mulder Th (2007) Motor imagery and action observation: cognitive tools for rehabilitation. J Neural Transm 114(10):1265–1278. https://doi.org/10.1007/s00702-007-0763-z
Mulder T, Zijlstra S, Zijlstra W, Hochstenbach J (2004) The role of motor imagery in learning a totally novel movement. Exp Brain Res 154(2):211–217. https://doi.org/10.1007/s00221-003-1647-6
Müller JFM, Orekhov Y, Liu Y, Ziemann U (2007) Homeostatic plasticity in human motor cortex demonstrated by two consecutive sessions of paired associative stimulation. Eur J Neurosci 25(11):3461–3468. https://doi.org/10.1111/j.1460-9568.2007.05603.x
Müller-Dahlhaus F, Lücke C, Lu MK, Arai N, Fuhl A, Herrmann E, Ziemann U (2015) Augmenting LTP-like plasticity in human motor cortex by spaced paired associative stimulation. PLoS ONE 10(6):1–14. https://doi.org/10.1371/journal.pone.0131020
Naatanen R, Kujala T, Winkler I (2011) Auditory processing that leads to conscious perception: a unique window to central auditory precessing opened by the mismatch negativity and related responses. Psychophysiology 548:4–22. https://doi.org/10.1111/j.1469-8986.2010.01114.x
Naish KR, Houston-Price C, Bremner AJ, Holmes NP (2014) Effects of action observation on corticospinal excitability: muscle specificity, direction, and timing of the mirror response. Neuropsychologia 64:331–348. https://doi.org/10.1016/j.neuropsychologia.2014.09.034
Nash AJ, Williams CS (1982) Effects of preparatory set and task demands on auditory event-related potentials. Biol Psychol 15(1–2):15–31. https://doi.org/10.1016/0301-0511(82)90028-X
Oldfield R (1971) The assessment and the analysis of handedness: the edinburgh inventory. Neuropsychologia 9:97–113
Palmieri RM, Ingersoll CD, Hoffman MA (2004) The Hoffmann reflex: methodologic considerations and applications for use in sports medicine and athletic training research. J Athl Train 39(3):268–277
Pascual-Leone A, Nguyen KT, Cohen AD, Brasil-Neto JP, Cammarota A, Hallett M (1995) Modulation of muscle responses evoked by transcranial magnetic stimulation during the acquisition of new fine motor skills. J Neurophysiol 74(3):1037–1045
Pinardi M, Ferrari F, D’Alonzo M, Clemente F, Raiano L, Cipriani C, Di Pino G (2020) Doublecheck: a sensory confirmation is required to own a robotic hand, sending a command to feel in charge of it. Cognitive Neuroscience, In Press
Ranieri F, Coppola G, Musumeci G, Capone F, Di Pino G, Parisi V, Di Lazzaro V (2019) Evidence for associative plasticity in the human visual cortex. Brain Stimul 12(3):705–713. https://doi.org/10.1016/j.brs.2019.01.021
Rauschecker JP (2011) An expanded role for the dorsal auditory pathway in sensorimotor control and integration. Hear Res 271(1–2):16–25. https://doi.org/10.1016/j.heares.2010.09.001
Rauschecker JP, Tian B (2000) Mechanisms and streams for processing of “what” and “where” in auditory cortex. Proc Natl Acad Sci 97(22):11800–11806. https://doi.org/10.1073/pnas.97.22.11800
Rioult-Pedotti MS, Friedman D, Hess G, Donoghue JP (1998) Strengthening of horizontal cortical connections following skill learning. Nat Neurosci 1(3):230–234. https://doi.org/10.1038/678
Rogasch NC, Fitzgerald PB (2013) Assessing cortical network properties using TMS-EEG. Hum Brain Mapp 34(7):1652–1669. https://doi.org/10.1002/hbm.22016
Romano-Smith S, Wood G, Wright DJ, Wakefield CJ (2018) Simultaneous and alternate action observation and motor imagery combinations improve aiming performance. Psychol Sport Exerc 38:100–106. https://doi.org/10.1016/j.psychsport.2018.06.003
Ronsse R, Puttemans V, Coxon J, Goble D, Wagemans J, Wenderoth N, Swinnen S (2011) Motor learning with augmented feedback: modality-dependent behavioral and neural consequences. Cereb Cortex 21(6):1283–1294. https://doi.org/10.1093/cercor/bhq209
Rosenkranz K, Williamon A, Rothwell JC (2007) Motorcortical excitability and synaptic plasticity is enhanced in professional musicians. J Neurosci 27(19):5200–5206. https://doi.org/10.1523/JNEUROSCI.0836-07.2007
Rosenkranz K, Kacar A, Rothwell J (2007) Differential modulation of motor cortical plasticity and excitability in early and late phases of human motor learning. J Neurosci 27(44):12058–12066. https://doi.org/10.1523/JNEUROSCI.2663-07.2007
Rossi S, Hallett M, Rossini PM, Pascual-Leone A (2009) Safety, ethical considerations, and application guidelines for the use of transcranial magnetic stimulation in clinical practice and research. Clin Neurophysiol 120(12):2008–2039. https://doi.org/10.1016/j.clinph.2009.08.016
Rossi S, Hallett M, Rossini PM, Pascual-Leone A (2011) Screening questionnaire before TMS: an update. Clin Neurophysiol 122(8):1686. https://doi.org/10.1016/j.clinph.2010.12.037
Rossini PM, Burke D, Chen R, Cohen LG, Daskalakis Z, DiIorio R, Ziemann U (2015) Non-invasive electrical and magnetic stimulation of the brain, spinal cord, roots and peripheral nerves: basic principles and procedures for routine clinical and research application. An updated report from an I.F.C.N. Committee. Clin Neurophysiol 126(6):1071–1107. https://doi.org/10.1016/j.clinph.2015.02.001
Ruffino C, Papaxanthis C, Lebon F (2017) Neural plasticity during motor learning with motor imagery practice: review and perspectives. Neuroscience 341:61–78. https://doi.org/10.1016/j.neuroscience.2016.11.023
Sakamoto M, Muraoka T, Mizuguchi N, Kanosue K (2009) Combining observation and imagery of an action enhances human corticospinal excitability. Neurosci Res 65(1):23–27. https://doi.org/10.1016/j.neures.2009.05.003
Salih F, Khatami R, Steinheimer S, Hummel O, Kühn A, Grosse P (2005) Inhibitory and excitatory intracortical circuits across the human sleep-wake cycle using paired-pulse transcranial magnetic stimulation. J Physiol 565:695–701. https://doi.org/10.1113/jphysiol.2004.082040
Sares AG, Deroche MLD, Ohashi H, Shiller DM, Gracco VL (2020) Neural correlates of vocal pitch compensation in individuals who stutter. Front Hum Neurosci 14:1–16. https://doi.org/10.3389/fnhum.2020.00018
Schaffert N, Janzen TB, Mattes K, Thaut MH (2019) A review on the relationship between sound and movement in sports and rehabilitation. Front Psychol 10:244. https://doi.org/10.3389/fpsyg.2019.00244
Schmitz G, Mohammadi B, Hammer A, Heldmann M, Samii A, Münte TF, Effenberg AO (2013) Observation of sonified movements engages a basal ganglia frontocortical network. BMC Neurosci 14(1):32. https://doi.org/10.1186/1471-2202-14-32
Scholz DS, Wu L, Pirzer J, Schneider J, Rollnik JD, Großbach M, Altenmüller EO (2014) Sonification as a possible stroke rehabilitation strategy. Front Neurosci. https://doi.org/10.3389/fnins.2014.00332
Scholz DS, Rhode S, Großbach M, Rollnik J, Altenmüller E (2015) Moving with music for stroke rehabilitation: a sonification feasibility study. Ann N Y Acad Sci 1337(1):69–76. https://doi.org/10.1111/nyas.12691
Scholz DS, Rohde S, Nikmaram N, Bruckner H-P, Grobbach M, Rollnik JD, Altenmuller EO (2016) Sonification of arm movements in stroke rehabilitation - a novel approach in neurologic music therapy. Front Neurol 7:1–7. https://doi.org/10.3389/fneur.2016.00106
Scott MW, Emerson JR, Dixon J, Tayler MA, Eaves DL (2019) Motor imagery during action observation enhances automatic imitation in children with and without developmental coordination disorder. J Exp Child Psychol 183:242–260. https://doi.org/10.1016/j.jecp.2019.03.001
Shams L, Seitz AR (2008) Benefits of multisensory learning. Trends Cogn Sci 12(11):411–417. https://doi.org/10.1016/j.tics.2008.07.006
Sigrist R, Rauter G, Riener R, Wolf P (2013) Augmented visual, auditory, haptic, and multimodal feedback in motor learning: a review. Psychon Bull Rev 20(1):21–53. https://doi.org/10.3758/s13423-012-0333-8
Silverstein J, Cortes M, Tsagaris KZ, Climent A, Gerber LM, Oromendia C, Edwards DJ (2019) Paired associative stimulation as a tool to assess plasticity enhancers in chronic stroke. Front Genetics 10:1–10. https://doi.org/10.3389/fnins.2019.00792
Simos PG, Kavroulakis E, Maris T, Papadaki E, Boursianis T, Kalaitzakis G, Savaki HE (2017) Neural foundations of overt and covert actions. Neuroimage 152:482–496. https://doi.org/10.1016/j.neuroimage.2017.03.036
Sowman PF, Dueholm SS, Rasmussen JH, Mrachacz-Kersting N (2014) Induction of plasticity in the human motor cortex by pairing an auditory stimulus with TMS. Front Hum Neurosci 8:1–6. https://doi.org/10.3389/fnhum.2014.00398
Stefan K, Kunesch E, Cohen LG, Benecke R, Classen J (2000) Induction of plasticity in the human motor cortex by paired associative stimulation Brain. J Neurol 123(Pt 3):572–584. https://doi.org/10.1093/brain/123.3.572
Stefan K, Wycislo M, Classen J (2004) Modulation of associative human motor cortical plasticity by attention. J Neurophysiol 92(1):66–72. https://doi.org/10.1152/jn.00383.2003
Stefan K, Wycislo M, Gentner R, Schramm A, Naumann M, Reiners K, Classen J (2006) Temporary occlusion of associative motor cortical plasticity by prior dynamic motor training. Cereb Cortex 16(3):376–385. https://doi.org/10.1093/cercor/bhi116
Ste-Marie DM, Law B, Rymal AM, Jenny O, Hall C, McCullagh P (2012) Observation interventions for motor skill learning and performance: an applied model for the use of observation. Int Rev Sport Exerc Psychol 5(2):145–176. https://doi.org/10.1080/1750984X.2012.665076
Sun Y, Wei W, Luo Z, Gan H, Hu X (2016) Improving motor imagery practice with synchronous action observation in stroke patients. Top Stroke Rehabil 23(4):245–253. https://doi.org/10.1080/10749357.2016.1141472
Suppa A, Li Voti P, Rocchi L, Papazachariadis O, Berardelli A (2015) Early visuomotor integration processes induce LTP/LTD-like plasticity in the human motor cortex. Cereb Cortex 25(3):703–712. https://doi.org/10.1093/cercor/bht264
Suppa A, Quartarone A, Siebner H, Chen R, Di Lazzaro V, Del Giudice P, Classen J (2017) The associative brain at work: evidence from paired associative stimulation studies in humans. Clin Neurophysiol 128(11):2140–2164. https://doi.org/10.1016/j.clinph.2017.08.003
Tanaka S, Kirino E (2018) The parietal opercular auditory-sensorimotor network in musicians: a resting-state fMRI study. Brain Cogn 120:43–47. https://doi.org/10.1016/j.bandc.2017.11.001
Taube W, Mouthon M, Leukel C, Hoogewoud HM, Annoni JM, Keller M (2015) Brain activity during observation and motor imagery of different balance tasks: An fMRI study. Cortex 64:102–114. https://doi.org/10.1016/j.cortex.2014.09.022
Terao Y, Ugawa Y (2002) Basic mechanisms of TMS. J Clin Neurophysiol 19(4):322–343. https://doi.org/10.1097/00004691-200208000-00006
Ticini L, Schutz-Bosbach S, Weiss C, Casile A, Waszak F (2011) When sounds become action: higher-order representation of newly learned action sounds in the human motor system. J Cogn Neurosci 24(2):464–474. https://doi.org/10.1162/jocn
Vogt S, Di Rienzo F, Collet C, Collins A, Guillot A (2013) Multiple roles of motor imagery during action observation. Front Hum Neurosci 7:1–13. https://doi.org/10.3389/fnhum.2013.00807
Vuckovic A, Osuagwu BA (2013) Using a motor imagery questionnaire to estimate the performance of a Brain-Computer Interface based on object oriented motor imagery. Clin Neurophysiol 124(8):1586–1595. https://doi.org/10.1016/j.clinph.2013.02.016
Williams SE, Cumming J, Ntoumanis N, Nordin-Bates SM, Ramsey R, Hall C (2012) Further validation and development of the movement imagery questionnaire. J Sport Exerc Psychol 34:621–646. https://doi.org/10.1123/jsep.34.5.621
Wolpert DM, Flanagan JR (2016) Computations underlying sensorimotor learning. Curr Opin Neurobiol 37:7–11. https://doi.org/10.1016/j.conb.2015.12.003
Wright DJ, Wood G, Eaves DL, Bruton AM, Frank C, Franklin ZC (2018) Corticospinal excitability is facilitated by combined action observation and motor imagery of a basketball free throw. Psychol Sport Exerc 39:114–121. https://doi.org/10.1016/j.psychsport.2018.08.006
Ziemann U, Muellbacher W, Hallett M, Cohen LG (2001) Modulation of practice-dependent plasticity in human motor cortex. Brain 124:1171–1181. https://doi.org/10.1093/brain/124.6.1171
Ziemann U, Ilić TV, Pauli C, Meintzschel F, Ruge D (2004) Learning modifies subsequent induction of long-term potentiation-like and long-term depression-like plasticity in human motor cortex. J Neurosci 24(7):1666–1672. https://doi.org/10.1523/JNEUROSCI.5016-03.2004
Conflict of interest
The authors declare no conflict of interests
Research involving human participants and/or animals
This study was carried out on human participants. Ethic approval was given by Brunel University ethic committee, and was in line with the declaration f Helsinki, and later amendments.
Prior to the beginning of data collection, participants read and signed an informed consent. They also consented to publish their data, in anonymised form.
Communicated by Francesca Frassinetti.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Below is the link to the electronic supplementary material.
About this article
Cite this article
Castro, F., Osman, L., Di Pino, G. et al. Does sonification of action simulation training impact corticospinal excitability and audiomotor plasticity?. Exp Brain Res 239, 1489–1505 (2021). https://doi.org/10.1007/s00221-021-06069-w
- Paired associative stimulation
- Transcranial magnetic stimulation
- Sensory augmentation