Myoelectric prosthetic devices are commonly used to help upper limb amputees perform activities of daily living, however amputees still lack the sensory feedback required to facilitate reliable and precise control. Augmented feedback may play an important role in affecting both short-term performance, through real-time regulation, and long-term performance, through the development of stronger internal models. In this work, we investigate the potential tradeoff between controllers that enable better short-term performance and those that provide sufficient feedback to develop a strong internal model. We hypothesize that augmented feedback may be used to mitigate this tradeoff, ultimately improving both short and long-term control. We used psychometric measures to assess the internal model developed while using a filtered myoelectric controller with augmented audio feedback, imitating classification-based control but with augmented regression-based feedback. In addition, we evaluated the short-term performance using a multi degree-of-freedom constrained-time target acquisition task. Results obtained from 24 able-bodied subjects show that an augmented feedback control strategy using audio cues enables the development of a stronger internal model than the filtered control with filtered feedback, and significantly better path efficiency than both raw and filtered control strategies. These results suggest that the use of augmented feedback control strategies may improve both short-term and long-term performance.
Recent advances in material design, micromachining, and the understanding of human neuromuscular systems have enabled the development of lightweight prosthetic devices that can be used to help amputees perform activities of daily living. One approach to controlling these devices is to use myoelectric signals sensed from contractions of the amputee’s remnant or congenitally different muscles1. Researchers have developed many signal processing techniques2, feature extraction methods3,4,5, and control strategies6 to enhance the performance of this approach. Despite these advancements in the field of myoelectric prostheses, many amputees still abandon their devices out of frustration7, due in part to insufficient precision in the control of prosthesis movements and a lack of adequate sensory feedback8,9.
Although invasive feedback, such as stimulation of sensory peripheral nerves10, has the potential to elicit close-to-natural tactile sensations, many prosthesis users prefer non-invasive feedback methods that do not require surgical intervention11. With this preference in mind, researchers have proposed using non-invasive sensory substitution methods to provide sensory information to prostheses users either through different sensory channels or using different modalities12. Vibro-tactile13, mechano-tactile14, electrotactile15,16,17, skin stretch18, and auditory19 are just some of the techniques that have been developed and used to provide prosthesis users with feedback. Although some studies (e.g.14,20,21) have shown that sensory feedback improves performance, others (c.f.14) have concluded that sensory feedback had no effect on performance. This lack of consensus arises, at least in part, because of an unclear understanding of how the incorporation of feedback relates to performance.
The role of feedback for real-time regulation and in improving human understanding of the control system and the task being performed is still unclear. Researchers have hypothesized that the human central nervous system implements control by estimating the current state of the musculoskeletal system and updating this estimate using sensory feedback22,23. This state estimation and prediction process is governed by a model formed in the central nervous system, which is known as internal model. This internal model holds properties of the arm, which are used to imitate its behavior24, predicts consequences of an action, and computes an action based on desired consequences25. Hence, this internal model is used in the feedforward control26 of the arm or residual limb, and affects the overall performance of a prosthesis27.
Humans use visual feedback along with other sensory and proprioceptive information to develop their internal model, however constant visual attention and the high level of concentration it requires may lead powered prosthesis users to reject their devices28. Conveniently, augmented feedback can be used to convey artificial proprioceptive and exteroceptive information21,29, which may help to develop strong internal models. Researchers have used audio augmented feedback in both robotic teleoperation30,31 and Brain Computer Interfaces (BCI)32 and have concluded that audio augmented feedback improves performance. Unlike visual feedback, audio requires less focus of attention and reduces distraction33,34. We hypothesize that audio augmented feedback may similarly improve the performance of myoelectric prosthesis control by enabling the development of stronger internal models.
In a recent study35, we assessed the performance and the strength of internal models developed by users when using two myoelectric control strategies that differed in control signals as well as level of feedback. The main aim of that study was to investigate whether the ability of users to adapt is influenced by the degree of sensory feedback inherent in different myoelectric controllers. Results from that study suggest that control strategies with raw control signals and high feedback level (Raw Control with Raw Feedback (RAW), such as with regression-based control) enable the development of a strong internal model, but at the expense of short-term performance. In addition, we found that control strategies with filtered control signals and reduced feedback (Filtered Control with Filtered Feedback (FLT), such as classification-based control) may enable better short-term performance, but hinder the development of internal models.
To mitigate this tradeoff, we have extended this work by decoupling the concepts of control and feedback through the use of augmented feedback. We combined the filtered control strategy that resulted in better short-term performance with audio augmented feedback from the raw control strategy that enables the development of stronger internal models. In this work, we assessed internal model strength and short-term performance for this audio augmented feedback control strategy known as Filtered Control with audio Augmented Feedback (AUG) and compared its results to the two commonly used myoelectric control strategies, RAW and FLT, that we assessed previously35. Our results show that the audio augmented feedback control strategy produces better short-term performance (as assessed using path efficiency and accuracy) than the feedback-rich control strategy, while enabling the development of a stronger internal model than the reduced feedback control strategy.
To inform the development of myoelectric control strategies with better short- and long-term performance, we investigated whether augmented feedback could improve user’s internal model strength without reducing short-term performance of the control. Short-term performance and internal model strength (as a predictor for long-term performance36) were evaluated while using a filtered control strategy that was augmented by audio feedback in a virtual target acquisition constrained-time task.
To assess the internal model developed by the human central nervous system and evaluate performance of the audio-augmented controller, we employed the same five parameters that were used in our previous work35:
Adaptation rate was extracted by computing the rate of feedforward modification of the control signal from one trial to the next37,38. The first 100–230 msec window of activations that started on the mark of the visual rendering on the screen for each trial was used to ensure that only the subject’s feedforward intent was captured and to avoid the effect of feedback synchronization. The target control signal was the activation of the cursor movement to the right direction only i.e., wrist extension. Other activations were considered as self-generated errors, which subjects were instructed to minimize.
The just-noticeable-difference (JND) parameter was computed from a perception threshold test as the threshold value reached after 23 reversals calculated by an adaptive staircase39. This parameter is influenced by both controller and sensory noise.
Internal model uncertainty P param was calculated using sensory uncertainty, controller uncertainty, JND, and adaptation rate35.
Performance of the audio-augmented control strategy was evaluated using path efficiency, computed by comparing the path taken to reach a target to the shortest Manhattan path to that target. Targets were defined by their location with respect to the X and Y axes on the computer screen. Targets that were located on either the X-axis or the Y-axis were known as on-axis targets, which only required the activation of a single DOF to be acquired. By the same token, targets that were not located on either the X-axis or the Y-axis were known as off-axis targets and therefore requiring the activation of 2 DOFs to be acquired (Appendix Figure A1).
Accuracy of target acquisition was computed as the Manhattan distance between the center point of a target and the actual final point reached. The ratio between this value and the optimal Manhattan path was then used to compute normalized accuracy as a percentage.
The adaptation rate serves as an indicator of how much humans modify their internal models when performing a task. A value of 1 indicates perfect adaptation; higher and lower values correspond to over- and under-compensation, respectively. Results for the adaptation rate test show that there was a statistically significant difference in adaptation rates to self-generated errors between subjects when using the RAW, FLT, and AUG controllers, as determined by a one-way ANOVA (F (2, 21) = 9.17, p = 0.02). A Tukey HSD post hoc test reveals that the adaptation rate for subjects using FLT (0.46 ± 0.19) was significantly lower than RAW (0.98 ± 0.3, p = 0.02) and AUG (0.86 ± 0.27, p = 0.021). In addition, no significant difference was found between adaptation rate data for subjects using the RAW and AUG control schemes (p = 0.65). These results suggest that audio-augmented feedback may enable similar adaptation behavior to feedback-rich control strategies (Fig. 1a).
The JND parameter is a measure of the smallest measure of stimulus that a subject is able to identify when using a certain controller. The lower this parameter, the better the ability of a subject to detect changes in the controller used. Like the adaptation rate results, analysis of JND test data using one-way ANOVA revealed a significant difference between subjects using RAW, FLT, and AUG (F (2, 21) = 5.17, p = 0.008). Upon running a Tukey HSD post hoc test on the data, we found that subjects who tested FLT had significantly higher JND values (59 ± 16 degrees) than other subjects who tested RAW (45 ± 20 degrees) and AUG (47 ± 13 degrees), but no significant difference in JND data between subjects who tested RAW or AUG (p = 0.92), which suggests that feedback in both RAW and AUG was sufficient to allow subjects to perceive a lower sensory threshold than the reduced feedback FLT (Fig. 1b).
Both adaptation rate and JND results were used to compute the internal model uncertainty for each of the tested control strategies. The lower the internal model uncertainty, the more confident a subject is in the control system used. Results showed significant difference in internal model uncertainty between the control strategies (robust Welch ANOVA (F (2, 13) = 9.3, p = 0.003)). Games-Howell post hoc analysis revealed a significant difference between internal model uncertainty for subjects who tested FLT (0.14 ± 0.1) and the subjects that tested RAW or AUG (0.055 ± 0.09, p = 0.035 and 0.041 ± 0.034, p = 0.004, respectively). On the other hand, there was no significant difference in the internal model uncertainty between subjects that used RAW or AUG (p = 0.56) (see Fig. 1c). These findings suggest that a strong internal model may be developed when using either the RAW or AUG control strategy, which confirms the first part of our hypothesis that audio-augmented feedback may enable the development of a stronger internal model than FLT alone.
For on-axis path efficiency, there was a significant difference between control strategies as determined by a one-way ANOVA (F (2, 21) = 5.8, p = 0.01). Bonferroni post hoc analysis showed that subjects who tested the AUG control strategy had significantly higher path efficiency than subjects who tested either the FLT or RAW strategies (p = 0.019, p = 0.02) (Fig. 2a). Analysis of the accuracy data collected from subjects using the three control strategies showed a significant difference between control strategies (one-way ANOVA (F (2, 21) = 4.5, p = 0.024)). Even though subjects that used FLT achieved high accuracy for on-axis targets (73 ± 7%), it was not significantly different (Bonferroni post hoc test) from subjects using who used RAW (65 ± 11%, p = 0.36) or AUG (80 ± 11%, p = 0.51). Conversely, subjects who tested AUG achieved significantly higher accuracies than those who tested RAW (p = 0.02) (Fig. 2b). These results further support our hypothesis that augmented feedback may also enable better short-term performance. For off-axis target performance tests, the null-hypothesis was not rejected when comparing path efficiency or accuracy data between the three control strategies (one-way ANOVA (F (2, 21) = 0.16, p = 0.86)) (Fig. 2c,d).
We ran 28 dependent paired t-tests to investigate possible learning effects when testing with one control strategy and retesting using another one (summarized in Appendix). Unlike adaptation rate, a significant learning effect was observed for JND values of subjects who used FLT after being exposed to AUG, where the JND dropped from 74 ± 8.0 degrees to 43 ± 4.2 degrees (p = 0.029). This suggests that there may be possible enhancement in sensory perception threshold when using the FLT control strategy after first being exposed to the feedback-richer AUG control strategy. On the other hand, learning effect for JND values for subjects who tested FLT after AUG were not significantly reflected in the internal model uncertainty (t (3) = 1.41, p = 0.25), because internal model uncertainty is affected by several other parameters e.g., adaptation rate and controller noise. These results suggest that internal model uncertainty for FLT may be improved after being first exposed to AUG by improving sensory perception, and less so by adaptation. Interestingly, on-axis target path efficiency significantly increased when testing the AUG control strategy after testing the feedback-rich RAW control strategy (t (3) = −14, p = 0.005). Also, accuracy for on-axis targets when using FLT after first being exposed to AUG increased significantly from 84 ± 3.7% to 94 ± 3.1% (t (3) = 23, p = 0.028).
Significant progress made in the fields of signal processing, sensory substitution and pattern recognition for myoelectric prostheses has allowed for improvements in the performance of myoelectric control. It stands to reason that such improvement is partially driven by feedback40, however contradictory results about the effect of feedback (in presence of vision) on performance oppose this belief41.
Several studies have investigated the effect of various feedback modalities on performance12. In particular, results from studies providing electromyography (EMG) biofeedback either through visual42 or electrotactile20 feedback have showed promising improvement in performance, but have not investigated the effect of this feedback on the internal model. Other researchers36 have investigated the effect of internal models on performance and found evidence of improvement in performance through improvement in the internal model. In a recent study43, researchers have found that feedback improves short and long-term performance and credited this improvement to internal models. However, they were not able to quantify the improvement in internal models that influenced the improvement in performance. To address this deficiency, in a previous study35 we used a psychophysical framework to quantify the strength of internal models developed for two commonly used myoelectric prosthesis controllers44, namely regression-based and classification-based controllers. In this study, we extended these previous studies by exploring the effect of relaying EMG biofeedback through the less-focus demanding audio feedback to augment myoelectric controllers that enable better short term performance. We hypothesized that augmented feedback may enable stronger internal model generation and better short-term performance. Our results showed that an audio-augmented feedback control strategy enables the development of a significantly stronger internal model than a classification-like filtered control with filtered feedback control strategy. This audio-augmented feedback controller also resulted in significantly better path efficiency for a one DOF task than both the raw feedback and the filtered feedback controllers.
Unlike classification-based control methods, the use of a regression-based control strategy enables the extraction of more information that may be used for augmented feedback. In this work, the concept of a hybrid control strategy that combines the robustness of classification-based control and the rich feedback of regression-based control was introduced. The concepts of feedback and control were effectively decoupled by employing classifier-like control, while providing the user with audio-augmented feedback derived from the feedback-rich regression control scheme.
Adaptation rate results showed that the controller with audio-augmented feedback enabled better adaptation behavior than FLT while maintaining similar adaptation behavior to the feedback-rich RAW. Hence, we may conclude that audio-augmented feedback may contain sufficient and comparable information as the RAW, therefore enabling better understanding of the myoelectric system than FLT. Equally as important, JND results for sensory perception showed that subjects who used either RAW or AUG achieved lower sensory thresholds than those who used FLT, which again supports the conclusion that feedback inherent in AUG was as effective as the feedback in the feedback rich RAW control strategy, enabling the perception of lower sensory thresholds. These results combined with results from the assessment of internal model uncertainty confirm our hypothesis that audio-augmented feedback enables the development of a strong internal model, possibly leading to better long-term myoelectric control performance.
On-axis performance results showed that audio-augmented feedback helped subjects perform better with respect to path efficiency and accuracy when compared to FLT. This finding suggests that subjects are able to integrate the audio feedback and use it for real time regulation as well as for internal model development. We also found that when subjects were first trained with AUG and then used FLT, they retained the internal model developed for AUG and were able to achieve better on-axis accuracy. This observation raises a new question to be investigated in future work: how long can this stronger internal model be retained before reverting completely to a model developed by the FLT control strategy?
Despite significant improvement in performance for on-axis targets when using the audio augmented controller, no significant improvement for off-axis targets accuracy or path efficiency was found. However, statistically significant results for accuracy and path efficiency for off-axis targets may be found with larger cohorts targeted towards these metrics. Future work informed by this study could include investigating the effect of augmented feedback on the internal model strength and short-term performance when using other controllers, assessing the impact of other modalities of augmented feedback (such as vibrotactile), exploring combinations of augmented feedback (such as audio + vibrotactile, which might enable an even stronger internal model), investigating how long the human central nervous system is able to retain the improved internal model strength (following a protocol similar to other studies43), and finally, assessing the benefits of using augmented feedback for limb-different individuals by performing psychophysical tests using prosthetic devices. Future studies can extend this work to accommodate the integration of feedback in real-time regulation using a framework such as the Smith predictor45,46 or the adaptive Smith predictor26,47.
Some limitations to this study were: the learning effect due to prolonged exposure to the feedback-rich control strategy was not investigated, no amputees were included in this study, and only a one DOF task was used for the assessment of the internal models.
The benefits of using audio-augmented feedback for myoelectric prosthesis control, highlighted in this study, call for inclusion of audio feedback in myoelectric control training procedures. For example, myoelectric prosthesis users may be provided with audio feedback when training with a myoelectric controller to improve their internal model strength and then prompted to use this control strategy without this audio feedback during regular use. One might argue that providing audio feedback using an earpiece or headphones to prostheses users might be impractical or uncomfortable, but emerging technologies, such as those that transmit sound using bone conduction, may soon enable users to benefit from other types of similarly augmented feedback. Furthermore, it is not yet known how frequently this augmented feedback is need. It is possible that users could occasionally enable the audio-augmented feedback to re-enforce their understanding.
In order to conduct a fair comparison between the control strategy developed here and the ones assessed in35, Support Vector Regression (SVR) was selected as the machine learning algorithm for the AUG strategy48. Figure 3 shows a sample of raw signals sensed from muscles. These signals were processed using standard EMG processing techniques used in the field of prostheses49,50. Specifically, these signals were low-pass filtered at 450 Hz with a fifth-order Butterworth filter and high-pass filtered using a third-order filter at 20 Hz51. In addition, a notch filter was implemented using a 2nd-order Butterworth band-stop filter from 57 Hz to 63 Hz52. Data were sampled at a frequency of 1 KHz and time-domain features53,54 were extracted from these processed signals every 64 msec using 160 msec windows to train the SVR model.
For real-time control, the SVR model produced output every 16 msec using 160 msec windows51. The output from this model was directly mapped to the velocity of the cursor for simultaneous activation of 2 degrees-of-freedom (wrist extension/flexion and adduction/abduction), but only the highest activated DOF was used in FLT; therefore, only allowing sequential control. Similar to FLT, the AUG control strategy only allowed sequential control, but used audio feedback to convey simultaneous control information, making this controller a hybrid between classification-based control and regression feedback (Fig. 4).
Audio Augmented Feedback
For the AUG control strategy, audio feedback maps the simultaneous activation of the 2 DOF motion using 4 audibly distinct frequencies i.e., one frequency per direction, (see Fig. 5a). The amplitude of the tone is directly proportional to the SVR output, requiring no knowledge of the intended task. For instance, a subject using AUG to reach a point to the right of the cursor while activating both the right and up directions, but with a higher activation in the right direction will see the cursor moving to the right accompanying to that cursor movement, the subject will hear two tones simultaneously with frequencies of 500 Hz and 800 Hz with the 500 Hz tone being louder than the 800 Hz tone.
General Experimental Procedures and Protocol
Each subject sat comfortably in a chair approximately 80 cm away from a computer screen with their right arm relaxed in a restraint and used their left hand to press keys on a keyboard placed on a desk in front of them. Subjects were asked to wear a 15 mW Cobra headset with the volume set to a no more than 52.5 ± 3 dB for the whole period of the experiment, but were allowed to remove it during scheduled breaks between the testing block. During the testing blocks, subjects controlled a cursor on a computer screen using isometric muscle contractions sensed by an electrode array placed on their forearm to acquire targets (Fig. 5b). Targets were crosshairs surrounded by an imploding red circle. Using the same experimental protocol implemented in35, which was approved by the University of New Brunswick’s Research and Ethics Board (file number 2014–019), subjects used each control strategy to complete a series of test blocks in a specific order after accomplishing a training block.
To determine the gain for mapping the control signals to the cursor velocity on the computer screen, each subject was asked to control a brush to paint a screen35. Their ability to maximize the area covered by paint determined the appropriate gains for each DOF, i.e., 20 pixels/s for wrist adduction/abduction DOF. The training block consisted of three sets of targets that appeared at predetermined, but randomly ordered, positions. Each set consisted of 16 targets, which subjects were asked to acquire in less than 12 seconds, after which it would disappear and another target would appear. The first test block was used to test the adaptation to self-generated errors. In this block, subjects were asked to repeatedly acquire a single target on the horizontal axis over a set of 80 trials.
Before the start of each test block, subjects were given a 2 minutes break, in which they were allowed to stand, remove the headset, and stretch if needed. The JND perception threshold test, the second test block, is a psychometric measure of sensory threshold for perception of a sensory stimulus. In this test, subjects were presented with two trials where one of these trials was perturbed with a stimulus. Subjects were then asked to select the trial which they thought had the added stimulus by pressing 1 for trial 1 and 2 for trial 2 on a keyboard39. The added stimulus was calculated using an adaptive staircase with accuracy set to 0.8455. Finally, the last test block was a performance test where subjects were asked to acquire targets, arranged in 2 sets of 16 targets each. For the first set of targets, subjects were tasked with acquiring each target in less than 1.7 s, whereas they were only allowed 1.4 s for each target in the second target set.
A total of twenty-four healthy right hand dominant subjects (15 male, and 9 female; mean and SD of age: 25 ± 6 years) participated in the study. All participants had either normal or corrected-to-normal vision. Informed consent was obtained from subjects before conducting the experiment according to the University of New Brunswick’s Research and Ethics Board (REB 2014-019). Subjects were randomly assigned to one of 3 main groups (eight subjects each). Group 1 subjects tested RAW and 4 subjects from this group were randomly selected to retest using AUG, referred to as subgroup 2. Group 2 subjects tested FLT and 4 subjects from that group were randomly selected to retest using AUG (subgroup 4). Finally, group 3 subjects tested AUG and 4 subjects were selected at random to redo the experiment using RAW (subgroup 5), while the remaining 4 subjects were asked to redo the experiment using FLT, and were referred to as subgroup 6.
Data from Groups 1, 2, and 3 were used to assess differences in outcome measures between the control strategies tested. Additionally, data from subgroups 2, 4, 5, and 6 were used to investigate possible learning effects between the commonly used control strategies and the audio-augmented feedback control strategy (Fig. 6). Learning effects between the raw and filtered controllers, subgroups 1 and 3, were investigated in a previous work35, which suggested significant improvements in internal model strength and adaptation rate when using the filtered controller after being exposed to the raw controller, but no significant improvements were observed when using the raw controller after being exposed to the filtered controller.
The Statistical Package for the Social Science software SPSS (IBM Corp, Released 2016, IBM SPSS Statistics for Windows, Version 24.0. Armonk, NY: IBM Corp) was used to run Levene’s test on JND, adaptation rate, internal model uncertainty, and performance measure results to investigate homogeneity in variances of the data. If data variances were found to be homogenous, one-way ANOVAs were used to assess differences between outcome measures for the control strategies tested. If statistical significance was found, post-hoc analysis was performed using either the Bonferroni or Tukey HSD test and reported using the one with the highest confidence in the p-value56. Conversely, if data variances were found to be heterogeneous, robust Welch ANOVA was used instead, and followed by post-hoc analysis using Games-Howell test (Fig. 7). Paired two sample t-tests were used with data collected from subjects in the subgroups (equal sample size). All analyses used a significance criterion of α = 0.05 and error bars shown in data plots were based on the standard error of the mean (SEM)57.
Data is available upon request - email corresponding author.
Childress, D. S. Historical aspects of powered limb prostheses. Clin Prosthet Orthot 9, 2–13 (1985).
Parker, P. A., Stuller, J. A. & Scott, R. N. Signal processing for the multistate myoelectric channel. Proceedings of the IEEE 65, 662–674 (1977).
Stango, A., Negro, F. & Farina, D. Spatial correlation of high density emg signals provides features robust to electrode number and shift in pattern recognition for myocontrol. IEEE Transactions on Neural Systems and Rehabilitation Engineering 23, 189–198 (2015).
Englehart, K., Hudgins, B., Parker, P. A. & Stevenson, M. Classification of the myoelectric signal using time-frequency based representations. Medical engineering & physics 21, 431–438 (1999).
Phinyomark, A., Phukpattaranont, P. & Limsakul, C. Feature reduction and selection for emg signal classification. Expert Systems with Applications 39, 7420–7431 (2012).
Erik Scheme, P. & Kevin Englehart, P. Electromyogram pattern recognition for control of powered upper-limb prostheses: State of the art and challenges for clinical use. Journal of rehabilitation research and development 48, 643 (2011).
Biddiss, E. A. & Chau, T. T. Upper limb prosthesis use and abandonment: a survey of the last 25 years. Prosthetics and orthotics international 31, 236–257 (2007).
Clancy, E. A., Morin, E. L. & Merletti, R. Sampling, noise-reduction and amplitude estimation issues in surface electromyography. Journal of Electromyography and Kinesiology 12, 1–16 (2002).
Doeringer, J. A. & Hogan, N. Performance of above elbow body-powered prostheses in visually guided unconstrained motion tasks. IEEE Transactions on Biomedical Engineering 42, 621–631 (1995).
Raspopovic, S. et al. Restoring natural sensory feedback in real-time bidirectional hand prostheses. Science translational medicine 6, 222ra19–222ra19 (2014).
Clemente, F., D’Alonzo, M., Controzzi, M., Edin, B. B. & Cipriani, C. Non-invasive, temporally discrete feedback of object contact and release improves grasp control of closed-loop myoelectric transradial prostheses. IEEE Transactions on Neural Systems and Rehabilitation Engineering 24, 1314–1322 (2016).
Antfolk, C. et al. Sensory feedback in upper limb prosthetics. Expert review of medical devices 10, 45–54 (2013).
D’Alonzo, M. & Cipriani, C. Vibrotactile sensory substitution elicits feeling of ownership of an alien hand. PloS one 7, e50756 (2012).
Antfolk, C. et al. Artificial redirection of sensation from prosthetic fingers to the phantom hand map on transradial amputees: vibrotactile versus mechanotactile sensory feedback. IEEE transactions on neural systems and rehabilitation engineering 21, 112–120 (2013).
Kaczmarek, K. A., Webster, J. G., Bach-y Rita, P. & Tompkins, W. J. Electrotactile and vibrotactile displays for sensory substitution systems. IEEE Transactions on Biomedical Engineering 38, 1–16 (1991).
Green, A., Chapman, C., Kalaska, J. & Lepore, F. Sensory feedback for upper limb prostheses. Enhancing Performance for Action and Perception: Multisensory Integration, Neuroplasticity and Neuroprosthetics 69 (2011).
Gonzalez-Vargas, J., Dosen, S., Amsuess, S., Yu, W. & Farina, D. Human-machine interface for the control of multi-function systems based on electrocutaneous menu: application to multi-grasp prosthetic hands. PloS one 10, e0127528 (2015).
Wheeler, J., Bark, K., Savall, J. & Cutkosky, M. Investigation of rotational skin stretch for proprioceptive feedback with application to myoelectric systems. IEEE Transactions on Neural Systems and Rehabilitation Engineering 18, 58–66 (2010).
Gonzalez, J., Soma, H., Sekine, M. & Yu, W. Psycho-physiological assessment of a prosthetic hand sensory feedback system based on an auditory display: a preliminary study. Journal of neuroengineering and rehabilitation 9, 33 (2012).
Schweisfurth, M. A. et al. Electrotactile emg feedback improves the control of prosthesis grasping force. Journal of neural engineering 13, 056010 (2016).
Markovic, M., Karnal, H., Graimann, B., Farina, D. & Dosen, S. Glimpse: Google glass interface for sensory feedback in myoelectric hand prostheses. Journal of Neural Engineering 14, 036007 (2017).
Wolpert, D. M., Ghahramani, Z. &Jordan, M. I. An internal model for sensorimotor integration. Science 269, 1880–1882 http://www.learning.eng.cam.ac.uk/zoubin/papers/WolGhaJor95.pdf https://doi.org/10.1126/science.7569931 (1995).
Gazzaniga, M. S. The cognitive neurosciences (MIT press, 2004).
Jordan, M. I. Computational aspects of motor control and motor learning. Handbook of perception and action 2, 71–120 (1996).
Cisek, P. Internal models. In Encyclopedia of Neuroscience (Springer, 2009).
Jordan, M. I. Chapter 2 computational aspects of motor control and motor learning. In Heuer, H. & Keele, S. W. (eds.) Motor skills, vol. 2 ofHandbook of Perception and Action, 71–120 (Academic Press, 1996).
Imamizu, H. et al. Human cerebellar activity reflecting an acquired internal model of a new tool. Nature 403, 192, https://doi.org/10.1097/00008526-199601000-00003 (2000).
Atkins, D. J., Heard, D. C. & Donovan, W. H. Epidemiologic Overview of Individuals with Upper-Limb Loss and Their Reported Research Priorities. JPO Journal of Prosthetics and Orthotics 8, 2–11 (1996).
González, J. & Yu, W. Multichannel audio aided dynamical perception for prosthetic hand biofeedback. In Rehabilitation Robotics, 2009. ICORR 2009 . IEEE International Conference on, 240–245 (IEEE, 2009).
Apostolos, M. K., Zak, H., Das, H. & Schenker, P. S. Multisensory feedback in advanced teleoperations: benefits of auditory cues. In Applications in Optical Science and Engineering, 98–105 (International Society for Optics and Photonics, 1992).
Liu, P. R. & Meng, M.-H. Acoustic display for navigation in internet-based teleoperation. In Intelligent Robots and Systems, 2005.(IROS 2005) . 2005 IEEE/RSJ International Conference on, 4161–4165 (IEEE, 2005).
Tidoni, E., Gergondet, P., Kheddar, A. & Aglioti, S. M. Audio-visual feedback improves the bci performance in the navigational control of a humanoid robot. Frontiers in neurorobotics 8 (2014).
Sigrist, R., Rauter, G., Riener, R. & Wolf, P. Augmented visual, auditory, haptic, and multimodal feedback in motor learning: A review. Psychonomic Bulletin & Review 20, 21–53, https://doi.org/10.3758/s13423-012-0333-8 (2013).
Eldridge, A. Issues in auditory display. Artificial Life 12, 259–274, https://doi.org/10.1162/artl.2006.12.2.259 (2006).
Shehata, A. W., Scheme, E. J. & Sensinger, J. W. Evaluating internal model strength and performance of myoelectric prosthesis control strategies. IEEE Transactions on Neural Systems and Rehabilitation Engineering 1–1, https://doi.org/10.1109/TNSRE.2018.2826981 (2018).
Lum, P. S., Black, I., Holley, R. J., Barth, J. & Dromerick, A. W. Internal models of upper limb prosthesis users when grasping and lifting a fragile object with their prosthetic limb. Experimental Brain Research 232, 3785–3795, https://doi.org/10.1007/s00221-014-4071-1 (2014).
Johnson, R. E., Kording, K. P., Hargrove, L. J. & Sensinger, J. W. Adaptation to random and systematic errors: Comparison of amputee and non-amputee control interfaces with varying levels of process noise. PloS one 12, e0170473 (2017).
Shehata, A. W., Scheme, E. J. &Sensinger, J. W. The effect of myoelectric prosthesis control strategies and feedback level on adaptation rate for a target acquisition task. In Rehabilitation Robotics (ICORR ), 2017 International Conference on, 200–204 (IEEE, 2017).
Faes, L. et al. Small-sample characterization of stochastic approximation staircases in forced-choice adaptive threshold estimation. Attention, Perception, & Psychophysics 69, 254–262 (2007).
Childress, D. S. Closed-loop control in prosthetic systems: historical perspective. Annals of biomedical engineering 8, 293–303 (1980).
Cipriani, C., Zaccone, F., Micera, S. & Carrozza, M. C. On the Shared Control of an EMG-Controlled Prosthetic Hand: Analysis of User–Prosthesis Interaction. IEEE Transactions on Robotics 24, 170–184 (2008).
Dosen, S., Markovic, M., Somer, K., Graimann, B. & Farina, D. EMG Biofeedback for online predictive control of grasping force in a myoelectric prosthesis. Journal of NeuroEngineering and Rehabilitation 12, 55, https://doi.org/10.1186/s12984-015-0047-z (2015).
Štrbac, M. et al. Short-and long-term learning of feedforward control of a myoelectric prosthesis with sensory feedback by amputees. IEEE Transactions on Neural Systems and Rehabilitation Engineering 25, 2133–2145, https://doi.org/10.1109/TNSRE.2017.2712287 (2017).
Hahne, J. M., Markovic, M. &Farina, D. User adaptation in Myoelectric Man-Machine Interfaces. Scientific Reports 1–10, https://doi.org/10.1038/s41598-017-04255-x (2017).
SMITH, O. J. M. A controller to overcome dead time. ISA J. 6, 28–33, https://ci.nii.ac.jp/naid/10003093722/en/ (1959).
Tan, K., Lee, T. & Leu, F. Predictive pi versus smith control for dead-time compensation. ISA Transactions 40, 17–29 (2001).
Miall, R., Weir, D. J., Wolpert, D. M. & Stein, J. Is the cerebellum a smith predictor? Journal of motor behavior 25, 203–216 (1993).
Ameri, A., Kamavuako, E. N., Scheme, E. J., Englehart, K. B. & Parker, P. A. Support vector regression for improved real-time, simultaneous myoelectric control. IEEE Transactions on Neural Systems and Rehabilitation Engineering 22, 1198–1209, https://doi.org/10.1109/TNSRE.2014.2323576 (2014).
Parker, P., Englehart, K. &Hudgins, B. Myoelectric signal processing for control of powered limb prostheses. Journal of Electromyography and Kinesiology 16, 541–548, https://doi.org/10.1016/j.jelekin.2006.08.006, Special Section (pp. 541–610): 2006 ISEK Congress (2006).
Ortiz-Catalan, M., Hakansson, B. & Branemark, R. Real-time and simultaneous control of artificial limbs based on pattern recognition algorithms. IEEE Trans Neural Syst Rehabil Eng 22, 756–764, https://doi.org/10.1109/TNSRE.2014.2305097 (2014).
Scheme, E. et al. Motion normalized proportional control for improved pattern recognition-based myoelectric control. IEEE Transactions on Neural Systems and Rehabilitation Engineering 22, 149–157, https://doi.org/10.1109/TNSRE.2013.2247421 (2014).
Wilson, A. W., Losier, Y. G., Parker, P. A. & Lovely, D. F. A bus-based smart myoelectric electrode/amplifier: System requirements. IEEE Transactions on Instrumentation and Measurement 60, 3290–3299, https://doi.org/10.1109/TIM.2011.2159316 (2011).
Scheme, E. J. & Englehart, K. B. Validation of a selective ensemble-based classification scheme for myoelectric control using a three-dimensional fitts’ law test. IEEE Transactions on Neural Systems and Rehabilitation Engineering 21, 616–623, https://doi.org/10.1109/TNSRE.2012.2226189 (2013).
Farrell, T. R. & Weir, R. F. The optimal controller delay for myoelectric prostheses. IEEE Transactions on Neural Systems and Rehabilitation Engineering 15, 111–118, https://doi.org/10.1109/TNSRE.2007.891391 (2007).
Ernst, M. O. & Banks, M. S. Humans integrate visual and haptic information in a statistically optimal fashion. Nature 415, 429–433 (2002).
Engineering Statistics Handbook, http://www.itl.nist.gov/div898/handbook/prc/section4/prc473.htm (2003).
Krzywinski, M. & Altman, N. Points of significance: error bars. Nature methods 10, 921–922 (2013).
The authors thank all subjects that volunteered to participate in this study. Many thanks to Dan Blustein for his support and valuable discussions.
The authors declare no competing interests.
Publisher's note: Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
About this article
Cite this article
Shehata, A.W., Scheme, E.J. & Sensinger, J.W. Audible Feedback Improves Internal Model Strength and Performance of Myoelectric Prosthesis Control. Sci Rep 8, 8541 (2018). https://doi.org/10.1038/s41598-018-26810-w
- Springer Nature Limited
This article is cited by
Wrist speed feedback improves elbow compensation and reaching accuracy for myoelectric transradial prosthesis users in hybrid virtual reaching task
Journal of NeuroEngineering and Rehabilitation (2023)
Virtual reality can mediate the learning phase of upper limb prostheses supporting a better-informed selection process
Journal on Multimodal User Interfaces (2023)
Scientific Reports (2022)
Experimental Brain Research (2022)
Immersive augmented reality system for the training of pattern classification control with a myoelectric prosthesis
Journal of NeuroEngineering and Rehabilitation (2021)