Background

Amputation is a debilitating event with substantial physical, social, and psychological consequences [1], but lost motor functions can be partially restored using myoelectric prostheses [1,2,3]. Modern devices include advanced robotic hands with several degrees of freedom capable of performing multiple grasp types [3, 4], but they do not provide somatosensory feedback; hence, the reconstruction of the lost limb is not complete. Only two commercial hands provide simple feedback to the user (e.g., Ability hand [5] and Vincent hand [6]), but the feedback conveys only the grasping force, while the natural feedback from biological hands includes both exteroception and proprioception. Providing the artificial proprioceptive feedback to the prosthesis users is important, as it would allow them to perceive the configuration of their bionic limb without looking at it. The ability to “feel” the prosthesis movements can also facilitate the embodiment, thus promoting the acceptance and usability of the system [2, 7].

The methods and technologies for restoring missing sensory information and consequently enriching the prosthesis-user interaction have been the focus of recent research efforts [1, 8,9,10,11]. Artificial feedback can be conveyed either invasively [12,13,14,15], by activating peripheral nerves using electrical stimulation, or non-invasively [9, 16,17,18,19], by delivering mechanical or electrical stimulation to the skin of the residual limb. Both methods can provide different types of information (e.g., pressure [20], texture [12], slippage [21], and pain [13]), but the grasping force is still the most common choice for the feedback variable [9, 18, 19, 22, 23]. Artificial proprioceptive feedback, which conveys the position of the prosthesis joints (e.g., wrist rotation and hand aperture), has been rarely considered [17, 24,25,26,27] and is typically limited to the hand aperture [9, 15] while fewer studies have investigated wrist rotation feedback [24, 28].

Non-invasive methods for providing feedback typically rely on sensory substitution. In this approach, prosthesis variables are translated into stimulation profiles according to a predefined encoding scheme, and the stimulation is delivered to the participant. Most often, the hand aperture and wrist rotation are conveyed using parameter modulation, which means that the value of the feedback variable is associated with the stimulation intensity or frequency [29, 30]. An alternative approach is to employ spatial encoding, in which the movement of the prosthesis is intuitively represented as a tactile sensation moving across the residual limb [31,32,33,34,35].

Modern prostheses integrate several active degrees of freedom (DoFs) [1, 4], usually combining hand opening and closing with active wrist rotation. Therefore, to convey the full state of the prosthesis to the user, artificial feedback needs to transmit two feedback variables: one encoding the hand aperture and the other encoding the wrist rotation. However, only a few studies have proposed feedback methods that can convey both variables [35]. In addition, although the two DoFs can be controlled using a simple 2-channel myoelectric interface, this approach is slow and cognitively taxing because the user needs to manually switch between the active functions [1]. In this case, the preferred control method is pattern classification because it eliminates tedious switching by directly translating the user motion intention into prosthesis movements [2, 4, 36,37,38,39]. However, only a few studies have tested the integration of feedback on multiple DoFs with pattern classification [17, 35, 40, 41]. For example, Shehata, et al. [40] proposed augmented audio feedback to provide 2-DoF movements of a cursor using different sound frequencies. Audio feedback was then applied to control a real prosthesis by adjusting the hand aperture and thumb abduction/adduction during a virtual egg test [42]. Patel, et al. [17] assessed the multichannel electrotactile feedback to provide individual finger positions when controlling a dexterous robotic hand. Similarly, Garenfeld, et al. [35] proposed discrete proprioceptive feedback to encode 2-DoF movements of a cursor simulating a prosthesis in a target-reaching task using multipad electrodes.

In our recent study [43], a novel approach was proposed to convey wrist rotation via an array of 8 vibration motors placed around the forearm. The feedback variable was transmitted through a combination of spatial encoding and Gaussian interpolation of intensity to generate a sensation that smoothly moved around the forearm congruent with the rotation of the prosthesis. Importantly, the tests showed that the smoothness of the tactile sensation, determined by the standard deviation (σ) of the Gaussian, could be modulated without affecting the subjects’ ability to localize the sensation, determined by the mean (µ) of the Gaussian [43]. Based on these results, we assumed that the two parameters of Gaussian interpolation can be used to provide two proprioceptive feedback variables independently and simultaneously. Therefore, in the present study, we propose a novel 2-DoF encoding scheme in which the hand aperture and wrist rotation are mapped to the parameters σ and µ, respectively. This mapping produces intuitive feedback because the parameter µ moves the peak of sensation around the forearm congruently with wrist rotation, while σ maps the hand aperture to the spread of sensation (i.e., larger hand opening, more spread sensation). The advantage of this novel method is that it can lead to a compact feedback interface that can convey the full kinematic state of the prosthetic hand through a single array of eight vibromotors (hereafter, the new method is referred to as Compact feedback).

To assess the validity and effectiveness of this approach, we implemented the novel feedback method together with pattern classification into an embedded platform and used it for closed-loop control of the Hannes prosthesis [44, 45]. This novel strategy was compared to the conventional approach, where the feedback was provided using two vibromotor arrays (with eight and four motors, respectively), each dedicated to conveying a single DoF (Conventional feedback). Reducing the number of vibromotors in the feedback array is an important step in facilitating the integration of the feedback interface into the prosthesis socket. The experimental assessment included ten able-bodied participants and four transradial amputees, who performed a Target Achievement Control Test (TAC) [46] using pattern classification to control rotation and hand aperture, while the state of the prosthesis was provided to the participant through vibrotactile feedback using either the Compact or Conventional interface. In these tests, the prosthesis was hidden from view and the motor sound was blocked, so that the participants needed to rely only on vibrotactile sensations to control the prosthesis. Moreover, a control group comprising ten able-bodied participants underwent the same experiment while receiving different levels of incidental feedback (visual, auditory and no feedback) with vibrotactile stimulation deactivated. These additional tests were considered control conditions and they were conducted to investigate how informative was vibrational feedback in comparison to the incidental feedback sources. Depending on the specific context (e.g., prosthesis type, task), these sources might be present or absent during the prosthesis use in real-life conditions [47].

Materials and methods

Participants

Ten able-bodied participants (aged 27.3 ± 3.06 yrs., 6 males) were recruited as the control group and other ten able-bodied participants (aged 27.3 ± 2.97 yrs., 5 males) plus four transradial amputees (aged 53.3 ± 14.7 yrs., all males) comprised test group. All the participants had no prior experience with proprioceptive tactile feedback. The control group performed the experiment in control conditions (visual, auditory and no feedback). The test group performed the same experiment using the two methods to provide vibrational feedback (compact and conventional interface).

The amputees were expert users of hand prostheses with a single active DoF (Table 1). Before starting the experiment, the participants were informed of the protocol, and they signed an informed consent form. The experimental protocol was approved by the local ethics committee.

Table 1 Information about participants with amputation
Full size table

Experimental setup

The experimental setup (Fig. 1A) comprised the following components: (1) a standard laptop (DELL XPS 15, Intel Core i9 @2.60GHz, 32GB RAM) running Windows 10, a keyboard, and a 24’’ computer monitor, showing a user interface to collect EMG data and train the pattern recognition algorithm to decode the user movements (E-DATS, EMG Data Acquisition & Training Software [45, 48]); (2) a virtual reality (VR) environment simulating the Hannes prosthetic hand (Hannes) [44, 45, 48, 49] with three active DoFs (hand open/close, wrist pronation/supination, and wrist flexion/extension); (3) the interface to select the type of feedback (Compact vs. Conventional), configure the feedback parameters and select the number of vibromotors used; (4) Hannes prosthetic hand [44] with three active DoFs (hand open/close, wrist pronation/supination, and wrist flexion/extension); (5) twelve eccentric rotating mass (ERM) vibromotors (Vybronics, VC0625B001L) with a custom-made control unit to provide tactile feedback; (6) six EMG electrodes (Otto Bock, 13E200 = 50 AC) with a custom-made master board to record the data, establish the communication between a host PC, Hannes hand and the feedback control unit; and (7) headphones to cancel the noise of the Hannes’s motors during movements.

Fig. 1
figure 1

Experimental setup and the scheme of closed-loop control. A: Experimental equipment used; B: Testing session with an amputee; and C: Closed-loop control of Hannes hand and its virtual representation. The subject (3) was seated in front of a monitor wearing the armband with EMG sensors (3b) placed equidistantly around the right forearm, and vibromotors (3a) distributed equidistantly around the interior aspect of the right forearm. The virtual reality interface (2) showed the orientation of the target (2b) and the controlled hand (2c) to implement target achievement test. Importantly, during the test, the subjects controlled a real prosthesis (4). The graphical controls (2a) allowed setting the paramenters of the feedback scheme (5). A PC application (6) governed the training of the pattern recognition model and later controlled the real prosthesis, while the prosthesis state (wrist angle and hand aperture) was conveyed to the subject through vibrotactile feedback. The subject wore headphones (7) to block the incidental noise coming from the Hannes prosthesis (4). More details about the setup and the experimental protocol are provided in the text

Full size image

The master board (Fig. 1C-1b) communicates with the Hannes hand (Fig. 1C-4) and with the feedback controller (Fig. 1C-1a) using a CAN Bus protocol; it collects data from EMG sensors (Fig. 1C-3b) using an onboard ADC unit and connects to the laptop via Bluetooth (Fig. 1C-2). The master board implements nonlinear regression (NLR) to decode the subject’s motion intention and generate control commands for Hannes and VR hands after offline calibration [45, 48]. The algorithm was trained to recognize wrist pronation/supination and hand opening/closing, and activate the prosthesis motion accordingly. The prosthesis velocity was proportional to the strength of the muscle contraction, as estimated by averaging the mean absolute value of the EMG recorded over the six electrodes. The E-DATS software responsible for collecting EMG data and calibrating the NLR algorithm was developed using Labview and MATLAB 2020b (MathWorks, Inc., Natick, MA, USA) [45, 48]. The VR framework was developed using the Unity Development Suite and C# language.

Another master board collected data from the Hannes hand (joint angles, encoders, currents, and EMGs) and sent that information to the VR framework. The framework computed the control commands for the vibromotors, depending on the selected encoding, and transmitted the commands to the master board, which activated the feedback controller to generate the desired vibrotactile stimulation. Therefore, the real hand performed the movements decoded by the master board through the pattern recognition algorithm, while the virtual hand replicated the position of the real hand.

Six EMG sensors were placed circumferentially and equidistantly around the participant’s forearm, approximately 5 cm distal to the elbow (Fig. 1C-3a and 3b). Each electrode was enclosed in a plastic block with conductive pads that were in contact with the skin. The blocks were equidistantly fixed with Velcro to an elastic band, which was strapped around the forearm by applying a level of pressure that was sufficient to hold the electrodes securely in place while avoiding discomfort during prolonged use. The EMG data was recorded at a sampling frequency of 1 kHz with 16 bits of resolution, and the signals were hardware rectified.

Two vibromotor configurations were used for feedback delivery. In one condition (Conventional interface), 12 vibromotors were arranged in two arrays (the first array of eight vibromotors and the second array of four vibromotors), whereas in the other condition (Compact solution), only one array of eight vibromotors was used. The arrays were placed semi-circumferentially and equidistantly around the internal aspect of the participant’s forearm, approximately 15 cm distal to the elbow (first array), and 20 cm distal to the elbow (second array) (Fig. 1B). The diameter of each vibromotor was 6.3 mm with a thickness of 2.5 mm. Each vibromotor was covered with a soft silicon case (size 14.2 × 14.2 × 6.8 mm) to localize and absorb the radiating stimulation and avoid heating the skin during prolonged continuous vibration. The silicone case was fixed to the elastic band using Velcro to prevent slippage. The case had housing to place the vibromotor vertically, thereby bringing the side of the coin motor in contact with the skin, as a pilot test showed that this produced stronger and more focused sensations. The motors were placed at a minimum distance of 20 mm between them to facilitate the vibration localization. The elastic band was strapped to the participant with Velcro by applying pressure, which was sufficient to hold the motors securely in place but without masking the vibration sensation and/or constricting the blood flow. To check tightness, the participant was asked to rotate the forearm, and if the elastic band moved accordingly without slipping, the band was deemed tight enough. The first vibromotor in each array was placed on the volar side, whereas the last was positioned on the dorsal side, covering only half of the forearm (its internal portion). This placement was selected because the spatial extent of the vibrotactile interface matched the range of motion of the prosthesis wrist. For each vibromotor, the vibration frequency was set to 200 Hz [50], which is within the range of the maximum sensitivity of Pacinian corpuscles [52]. The gain of each vibromotor was adjusted individually for each participant (see section Experimental protocol).

The participants sat comfortably in front of the monitor, with their right arm (residual limb) relaxed and placed on a desk, ensuring that the vibromotors did not contact the table. The computer monitor was placed approximately 50 cm from the participant, and the Hannes hand was placed behind the screen so that the participant could not see it while wearing noise-canceling headphones to block the sound from the prosthesis motors. The VR environment displayed two virtual Hannes hands, where one showed the target position (Fig. 1C-2b, transparent Hannes), while the other was controlled by the participant (Fig. 1C-2c, solid Hannes). As explained above, the participant controlled the movements of the real prostheses through pattern recognition by moving their hand (able-bodied participants) or contracting their muscles to “move” the phantom limb (amputee participants). Therefore, the VR environment was employed to show the task to the participants (target configuration), while the real hand ensured that they controlled the real dynamics of a physical prosthesis.

Vibrotactile feedback encoding

A novel encoding scheme was implemented to provide proprioceptive feedback (Fig. 2, Compact feedback, CM) and was compared with standard spatial encoding (Fig. 2, Conventional feedback, CN). When using the latter, information regarding wrist rotation and hand aperture was provided using two arrays of vibromotors. Eight vibromotors were used to convey wrist rotation, while a separate array of four vibromotors provided the hand aperture, reflecting a larger range of motion in the wrist than in the hand. The range of motion of each DoF was divided into equisized intervals, which were then sequentially associated with the vibromotors. Therefore, as the wrist rotation changed from 180° to 0°, the vibromotors were activated sequentially from the motor on the dorsal side to the motor on the volar side of the forearm and similarly for the hand aperture. For instance, vibromotor 5 was activated when the wrist angle was 120° (Fig. 2B), and then as the wrist continued rotating, motors 4 and 3 were activated in succession when the wrist angle reached 100° and 80° (Fig. 2C), respectively, and analogous for the hand aperture (Fig. 2B and D).

Fig. 2
figure 2

Illustration of the two feedback encoding approaches. A1: Representation of hand aperture range of motion; A2: Representation of the wrist rotation range of motion; B, C and D: Vibromotor activations determined by the two encoding approaches for three different hand configurations (top). The Conventional feedback (middle) used two arrays of vibromotors to separately convey wrist rotation and hand aperture. The Compact feedback (bottom) employs a single array of vibromotors and transmits the feedback information by modulating the Gaussian mean (µ) for wrist orientation and standard deviation (σ) for hand aperture

Full size image

The novel encoding scheme (Compact) conveyed two feedback variables using a single array of eight vibromotors, and hence represented a more compact solution (four vibromotors less). This approach was based on our recent study [43], where we investigated the Gaussian interpolation of vibrotactile stimulation and proposed (but did not test) that the location and spread of the Gaussian profile could be perceived independently by the participants. Therefore, in the present study, the rotation angle of the prosthesis wrist was conveyed to the participant by modulating the vibration location (i.e., the location of the peak intensity, µ in Eq. 1), while the hand aperture was transmitted by varying the spread of sensations (i.e., the number of active motors, σ in Eq. 1), as shown in Fig. 2 (Compact feedback, see also below). The feedback was, therefore, intuitively related to prosthesis motion, as the rotation of the wrist produced a sensation that rotated around the forearm, while with the hand opening, the elicited sensation spread spatially in both directions from the current location.

The following equation was used to define the Gaussian mapping between the wrist position and hand aperture:

$$\:\left\{\begin{array}{c}\begin{array}{l}y_k={e}^{-\frac{{\left(k-\mu\:\right)}^{2}}{{g*\sigma\:}^{2}}}\\\:\mu\:=p*N\end{array}\\\:\sigma\:=\left(\frac{h}{100}*\left(2.0-0.1\right)\right)+0.1\end{array}\right.$$
(1)

where \(y_k\) is the normalized vibration amplitude of the motor \(\:k\), \(\:\mu\:\) is the mean of the Gaussian computed as the normalized position (\(\:p\)) of the wrist multiplied by the number of motors (\(\:N\) = 8) in the array, \(\:g\) is set to 2.25, and \(\:\sigma\:\) is the standard deviation of the Gaussian ranging from 0.1 for the fully closed hand (0%) to 2.0, for the fully open hand (100%), calculated as the normalized position of the hand (\(\:h\)) multiplied by the range of σ ([0.1 2.0]). The values and ranges of these parameters were selected based on the outcomes of our recent study [43]. The parameter \(\:\mu\:\), which was mapped to the wrist position, determined the location of the peak intensity within the array of vibration motors (increasing \(\:\mu\:\) means that the prosthesis is supinating), while \(\:\sigma\:\) was mapped to the hand aperture and adjusted the spread of intensity across the motors (increasing \(\:\sigma\:\) means that the prosthesis is opening).

An example of Compact encoding is shown in Fig. 2. The peak of intensity was centered at vibromotor 5 when the wrist angle was 120° (Fig. 2B), and then gradually moved across the vibromotors as the wrist continued rotating to reach the third vibromotor when the wrist angle was 80° (Fig. 2C). Regarding the hand aperture, three vibromotors were simultaneously active (with different intensities) when the hand aperture was 35% (Fig. 2B), and then more motors were added so that five vibromotors were active when the hand aperture was 75% (Fig. 2D). Figure 2 also shows the feedback pattern for the Conventional approach (middle row with two arrays), which is based on the discrete activation of individual motors, so that the two methods can be directly compared.

Experimental protocol

Test conditions

The participants were comfortably seated in front of a monitor placed at a distance of approximately 50 cm. The band with EMG electrodes and vibromotors was mounted on the forearm of the dominant hand (able-bodied participants) and/or residual limb (amputee participants). The minimum and maximum amplitudes for each vibromotor were determined using the method of limits [52] as described in our recent study [43]. Briefly, the vibration intensity was increased in small steps (i.e., 4–5% of maximum activation) until the participants reported a sensation or discomfort. During the experiment, the vibration intensity was modulated between the sensation and discomfort thresholds to generate clearly perceivable and localized sensations that were not intrusive.

Subsequently, the gain of the EMG electrodes was adjusted by visualizing the signals to ensure that maximum contraction did not saturate the recording. Subsequently, the EMG data used for classifier training were collected. To this end, the participant was asked to perform four different movements (hand closing and opening, wrist pronation, and supination) at three different contraction levels (weak, medium, and strong). Each movement was maintained for 2 s with a 5 s rest between repetitions. After data collection, the NLR classifier was trained, and its parameters were sent to the master control board for real-time decoding of participant movements and prosthesis control. The detected movement class activated the corresponding DoF of the prosthesis, whereas the average magnitude of the myoelectric signal determined the movement speed. The latter was additionally fine-tuned for each participant until they were satisfied with how the prosthesis reacted to their commands.

The participants then practiced prosthesis control, and when confident, proceeded to the feedback familiarization phase. In this phase, they were trained to associate prosthesis movements with vibromotor activation. The order of the feedback schemes (Conventional and Compact) was chosen pseudo-randomly so that half of the participants started with the Conventional (CN) configuration and half with the Compact (CM) configuration. The participants were asked to move the prosthesis one joint at a time five times across the full range of motion while they watched the prosthesis moving with haptic feedback active. They then commanded sequential movements around 2-DoFs to experience the vibration patterns related to such combined activations.

After the control and feedback familiarization, the participants performed the experiment. The experiment was based on a target achievement control test (TAC) in which the configuration of the controlled hand was adjusted to match that of the visual target shown on the computer screen. Importantly, the participants controlled a real prosthesis while the incidental feedback from the device (sound and vision) was blocked, as described previously. A sequence of 21 trials, hereafter named “block,” was generated by pairing 21 pseudo-random wrist targets (3 repetitions × 7 orientations) with 21 pseudo-random hand targets (5 repetitions × 4 levels of aperture + 1). The seven target orientations of the wrist and four target positions of the hand aperture were equally distributed between 0°-180° and 0-100% excluding the extremities, respectively (see Fig. 3). Moreover, to additionally challenge the participants and motivate them to rely on the feedback, the prosthesis velocity was changed across trials by multiplying the maximum speed (1.26 rad/s) by a gain randomly selected from the interval 0.4–1. The gains for the wrist and hand aperture were independently generated. At the beginning of each trial, the participant could see the starting configuration of the controlled hand visualized on the computer screen. Its position was selected to be at the end of the range of motion furthest from the target configuration (Fig. 3A, green dashed line), that is, if the target wrist orientation was over 90°, the wrist of the controlled hand would start at 0°. Similarly, if the target hand aperture was over 50%, the controlled hand would start fully closed (hand aperture of 0%).

Fig. 3
figure 3

Target positions and outcome measures. A: Error between target and reached position (control accuracy) when adjusting the wrist; B and C: Optimal and generated paths and end-point errors for hand and wrist, respectively. Trial time as well as optimal times (topt1 and topt2) to adjust each DoF are annotated on the x-axes.

Full size image

After a sound notification indicated that the trial started, the participants were asked to adjust the controlled hand using pattern classification control, while the state of the hand (wrist and hand position) was provided only by tactile feedback. The participants could control different DoFs as they wished, and once they judged that the desired target configuration was reached, they pushed the spacebar. The reached position was then revealed to them by showing the controlled hand on the screen, and after a short break (5 s), the next trial was started. After completing the block of 21 trials, a 5 min resting phase was provided, followed by another block of trials (three blocks in total). After ten minutes of rest, the second feedback scheme was tested following the same protocol.

At the conclusion of the experiment, the participants were asked to choose their preferred vibrational feedback scheme.

Control conditions

The participants in the control group followed the same training and feedback familiarization protocol described in the previous section for the participants in the test group, using the feedback defined by the respective condition. Then, they underwent control experiments where they did not receive any vibrotactile stimulation. The band with vibromotors was thus not placed around their forearms. The control group performed the experiment in three control conditions:

  • Visual feedback (VF): in this condition, the prosthesis was placed in front of the participants so that they could see it during the control.

  • Auditory feedback (AF): here, the prosthesis was hidden from view, but the participants were not wearing the noise cancelling headphones, and hence, they could hear the motor sound as the prosthesis moved (closing and rotation).

  • No feedback (NF): in this case, the participants could rely only on feedforward control and natural feedback (e.g., sense of muscle contraction).

The order of the three control conditions was randomly selected across participants. NF was included to assert that vibrotactile feedback was indeed used by the participants. If there was no difference in performance between NF vs. CN and CM, this would mean that vibrotactile feedback was in fact redundant (and hence completely unnecessary). VF was expected to lead to the best performance due to the dominance of visual feedback, especially when controlling something that can be observed (position vs. force control), and AF was included for completeness as the audition is also an important and highly sensitive feedback channel.

Data analysis

The primary outcome measure was the endpoint error between the target configuration and controlled hand (Fig. 3A). This measure was computed for each DoF as the difference between the target position and the position of the hand at the moment the participant pressed the space bar to indicate the end of the trial (Fig. 3B and C). The position errors were expressed as a percentage of the range of motion. In addition, for the given motion speed of the controlled hand in each trial, the path efficiency was calculated for each DoF as the ratio between the length of the optimal trajectory (Fig. 3B and C, yellow line) and the trajectory generated by the participant (Fig. 3B and C, green line). The optimal trajectory is the shortest path from the initial configuration to the target configuration. Instead, the generated trajectory was calculated as the sum of the lengths of the slope segments (Fig. 3B and C, green full lines). The horizontal segments (Fig. 3B and C, green dashed line) were not counted in the trajectory length because they corresponded to the phases in which the participant did not move the prosthesis.

Another parameter used to assess performance was the DoF time, that is, the amount of time that the DoF was activated during the trial, expressed as the percentage of the total trial time. Time efficiency was also calculated as the ratio between the optimal time to perform the trial for the given motion speed and the time taken by the participant to perform the trial (Fig. 3B and C, trial time). The optimal time was computed as the sum of the time to adjust the two DoFs (topt1 + topt2) following the respective optimal paths, because the participant was able to adjust them sequentially and not simultaneously. Note that, contrary to path efficiency, which was computed for each DoF separately, time efficiency was estimated per trial (because the rest time could not be separated into individual DoFs).

For able-bodied participants, the average of the outcome measures (error, path efficiency, DoF time, and trial time) was computed for each participant and condition and statistically compared between the conditions for each DoF. Data were tested for normality using the Shapiro-Wilk test. For each group of able-bodied participants (both test and control), a one-way ANOVA or Friedman test was conducted depending on the outcome of the normality test to compare the conditions performed by the specific group, namely, CM vs. CN for test and AF vs. VF vs. NF for control group. Then, in the latter case, post hoc pairwise comparisons were performed using a paired t-test or Wilcoxon signed rank test with Bonferroni correction. Additionally, a comparison between test conditions and control conditions (CM and CN vs. VF, AF, and NF) was performed using a two-sample t-test or Mann-Whitney U test depending on the outcome of the normality assessment. OriginPro 2020 Graphic & Analysis (OriginLab Corporation, Northampton, MA, USA) and MATLAB 2020b (MathWorks, Inc., Natick, MA, USA) were used for statistical analysis. The threshold for statistical significance was set at p < 0.05, and outliers were excluded from statistical comparisons. Data was considered an outlier if the value was more than 1.5 times the interquartile range below or above the 25th and 75th percentiles of the sample, respectively.

For the transradial amputees, the average of the outcome measures was computed for each DoF, feedback scheme, and participant, and reported overall and separately across the three blocks to assess potential learning. Furthermore, for each DoF and condition, the means of the three blocks were reported to assess and compare the two feedback schemes. For consistency, the results in the text are reported as the mean ± standard deviation in all cases.

Results

The overall results for error, path, and time efficiency for able-bodied participants in test (CN and CM) and control conditions (AF, VF, and NF) are summarized in Fig. 4. Overall, the differences between the conditions were mostly consistent for both DoFs and therefore the two DoFs are not discussed separately (except where this is explicitly mentioned).

Fig. 4
figure 4

Summary results for the positioning error (A), path (B) and time efficiency (D), and time per DoF (C) in the form of boxplots for able-bodied subjects and two feedback schemes (green – conventional [CN], purple – compact [CM]). HA indicates the aperture and WR indicates the wrist rotation. The small circles are the means, the red lines indicate the medians, boxes are interquartile ranges, whiskers represent min/max values and the crosses are outliers. Dashed vertical lines separate control (left – VF, AF and NF) and test (right – CN and CM) conditions. Horizontal bars show a statistically significant difference between the connected conditions (*, p < 0.05; **, p < 0.01). The asterisks above a condition indicate that the condition was significantly different from all others

Full size image

Regarding the accuracy of control, the visual feedback condition showed the best performance in terms of endpoint error (~ 5%), significantly outperforming all other conditions (Fig. 4A). Notably, no statistically significant differences emerged between the two vibrational feedback schemes (CN and CM) in error rate (endpoint error ~ 20%), although the performance appeared to be more consistent across participants when using compact configuration to adjust hand aperture (lower interquartile range, Fig. 4A for HA). Regarding subjective preference, 6 out of 10 able-bodied participants chose CN as the preferred approach. Importantly, both vibrotactile feedback approaches significantly outperformed the condition with no-feedback. The latter condition was characterized with the highest endpoint error (~ 30%) overall. Interestingly, the performance with auditory feedback was similar to that of vibrotactile feedback and even slightly but significantly better compared to CM when controlling hand aperture (endpoint error of 18.2 ± 3.4% vs. 21.2 ± 2.5%).

Regarding the path efficiency (Fig. 4B), there was no significant difference between the two vibrotactile feedback conditions. However, there are statistical differences between NF vs. VF and CM in both hand aperture and wrist rotation, and between NF and AF for hand aperture, and AF and VF for wrist rotation. Accordingly, the participants were more efficient when performing the task without any feedback (NF). This reflects the fact that in this condition, the participants relied mostly on feedforward control generating “straight” trajectories with no corrections, given the absence of any feedback information regarding the prosthesis configuration. Specifically, the path efficiency achieved in NF (~77%) was significantly higher than AF (67.9 ± 9.9%), VF (70.8 ± 10.9%) and CM (65.3 ± 15.5%) for hand aperture, and VF (69.4 ± 9.8%) and CM (60.8 ± 19.7%) for wrist rotation.

Differently, the DoF time parameter (Fig. 4C) exhibited statistically significant differences only for hand aperture, where the DoF time was lower for VF (~ 16%) compared to AF (~ 22%) and NF (~ 21%). It seems that, in general, the participants spent somewhat more time adjusting the wrist compared to the hand. Also, summing up the times for the two DoFs indicates that the participants actively controlled the prosthesis for only ~ 40% of the total trial time, whereas the rest was used presumably to decide about the next action (e.g., which command to send to the prosthesis).

Finally, in the time efficiency (Fig. 4D), there was no significant difference between the feedback conditions, except between NF vs. VF and AF. The vibrotactile feedback conditions, however, seem to exhibit higher variability across participants (i.e. by comparing the interquartile ranges between CN and CM vs. AF, VF and NF). Likewise, the time efficiency in NF (42.8 ± 11.2%) was significantly higher compared to VF (23.9 ± 5.9%) and AF (31.6 ± 6.3%). Importantly, taken together, the results indicate that no statistically significant difference was found between the two vibrotactile feedback types (CN vs. CM) in any of the considered measures.

Figure 5 shows the representative trajectories (green line) generated by an amputee participant when adjusting the configuration of the prosthesis using the two vibrotactile feedback encoding schemes (Fig. 5A for the Conventional and Fig. 5B for the Compact feedback). The target positions for wrist rotation and hand aperture are indicated by dark magenta lines. In both cases, the participant first adjusted one DoF fully and then proceeded to the next one, namely, wrist rotation and then hand aperture. However, in the trial with the Conventional feedback (Fig. 5A), the participant returned to readjust the wrist because he did not consider it to be completely correct. The control of wrist rotation took a longer time and included more feedback-driven corrections (flat segments) than the hand aperture, which was adjusted in one (Fig. 5A) and two (Fig. 5B) continuous motions. This observation was the same for both feedback conditions. In two cases (wrist in Fig. 5A and hand in Fig. 5B), the participant reached the end of the range of motion and used this as a well-defined and easy-to-recognize “anchor” point, from which they then moved the prosthesis to the target position. Finally, with both feedback schemes, the end positions achieved in both DoFs closely matched the target values. Figure 5 also shows the outcome measures for these specific trials to illustrate their range and meaning.

Fig. 5
figure 5

Two example trials performed by an amputee participant using the two feedback methods compared in the manuscript, namely, Conventional (A), and Compact Feedback (B). The plots show the trajectories generated by the participant when adjusting the wrist rotation and hand aperture. Err, PathEff, TimeEff and DoFTime denote positioning error, path and time efficiency, and time per DoF, while the subscripts HA and WR denote hand and wrist, respectively

Full size image

The results for the amputees are shown individually for each participant and block of trials in Fig. 6. In general, the amputee performance (Fig. 6) was comparable to that obtained in the able-bodied participants, except for P1, who obtained better accuracy (Fig. 6A and B) but worse results in all efficiency (Fig. 6C, D, G and H) parameters. Again, there is a trend for DoF time, where a longer time was invested to adjust the wrist compared to the hand, regardless of the feedback scheme (Fig. 6E and F). When comparing Conventional and Compact feedback schemes, neither showed a clear advantage, thereby reflecting the overall results obtained for able-bodied participants. Overall, the difference in performance ([min, max]) between the encoding schemes across participants was not large in any case; specifically, [0.2%, 9.7%] for error (Fig. 6A and B), [0.5%, 18.6%] for path efficiency (Fig. 6C and D), [0.6%, 17.0%] for DoF time (Fig. 6E and F), and [2%, 9.3%] for time efficiency (Fig. 6G and H). The scheme that led to a lower error depended on the participant and even on the degree of freedom.

Fig. 6
figure 6

The results obtained by 4 amputee participants using Compact and Conventional feedback. Left panels show the performance for each block of trials, to assess the potential learning effect, while the right panels represent the average across blocks. HA indicates aperture, and WR represents wrist rotation. Different colours are associated to different participants (P1-P4). The horizontal dashed lines in the right pannels indicate the mean performance of able-bodied subjects. Bn in the left plots denote different testing blocks

Full size image

There seems to be a slight trend for better efficiency of the Compact solution, as it is higher in most cases. Among participants individually, P1 was the least efficient, but also made the smallest errors in most cases, whereas P3 was the most efficient while maintaining good accuracy. Across blocks, there is no consistent learning across participants and conditions but only sporadic tendencies. For instance, P2 and P3 decreased, whereas P4 somewhat increased the error from the first to the third block.

Finally, P1 and P3 preferred the Compact configuration, whereas P2 and P4 preferred the Conventional configuration; therefore, the participants generally preferred the interface with which they achieved better accuracy.

Discussion

The compact feedback solution allows intuitive mapping of multiple sensory information

The present study proposes a novel approach to convey proprioceptive information simultaneously for two feedback variables, namely, hand aperture and wrist rotation, by using a single array of vibration motors. The approach employs spatial encoding with Gaussian interpolation, in which the location of the peak intensity is modulated to convey wrist orientation (\(\:\mu\:\) in Eq. 1), while the spread of sensation (\(\:\sigma\:\) in Eq. 1) is associated with the hand aperture. Most methods in the literature use multiple arrays of stimulators to provide information from different DoFs [53, 54], whereas the encoding proposed in the present study enables conveying multiple information streams through a single array. The latter is an important strategy as it makes the feedback interfaces more compact and therefore easier to integrate into the prosthetic socket. Approaches to establish parallel feedback channels through the same interface have been investigated in the literature using multimodal (hybrid vibro- and electrotactile stimulation) [31,32,33,34] and multiparameter (amplitude and frequency) [8, 9, 29, 30] modulations. Our approach is based on spatial encoding, which enables us to superimpose two feedback variables using intuitive mapping, that is, a rotating sensation to convey rotation and a spreading sensation to indicate hand opening.

Importantly, the novel approach based on Compact encoding was compared to Conventional mapping that used two independent arrays, and the results demonstrated that the two approaches were similarly effective in conveying two-DoF proprioceptive information. There was no significant difference in either accuracy or efficiency when the two encoding schemes were used for closed-loop prosthesis control.

The present study has therefore proven the hypothesis inspired by a recent study from our group that the two parameters defining the Gaussian interpolation strategy can be independently modulated and recognized by participants [43]. In that study, the participants controlled only the wrist rotation of a virtual prosthesis, whereas the parameter \(\:\sigma\:\) was imposed externally as “noise.” In the present experiment, however, they actively adjusted the configuration (both wrist and hand) of a real prosthesis. Note that feedback encoding in the Compact interface is intrinsically coupled. For instance, the same target hand aperture leads to different sensations, depending on the target wrist rotation. This effect does not exist in the Conventional approach, and coupling, therefore, does not negatively affect control.

Importantly, the amputee participants achieved a performance that was subject-specific but coherent with that of the able-bodied population (Fig. 6). This is an important outcome for the prospective clinical application of the proposed closed-loop control, especially considering that the amputee participants employed pattern classification control and a feedback interface, which were mounted on the same forearm. This setup can be used in clinical applications.

Overall, the novel method allowed for effective closed-loop control (average error ~ 20%) in both able-bodied and amputee participants. However, the average error was higher than that obtained in a recent study in which Gaussian interpolation was used to adjust the wrist orientation [43]. Nevertheless, in that study, the participants controlled a virtual prosthesis using a keyboard; hence, both the system and the control interface were ideal. We also assume that in clinical applications, precise control of wrist orientation and hand aperture will not be critical. Both wrist rotation and hand opening can be adjusted with a margin of error without jeopardizing successful grasp. In addition, closed-loop control is expected to improve with prolonged training because of better use of pattern classification and more reliable interpretation of the feedback. Similarly, prolonged use could also change subjective preferences, but this remains to be tested.

In general, the participants spent more time adjusting the wrist than the hand. This is not unexpected, as the wrist has a larger range of motion, more target and feedback levels, and adjusting the wrist is, therefore, a more complicated task. However, it is still an encouraging result that despite this difference, both DoFs can be controlled with similar relative accuracy. In general, the time efficiency was not high, which is likely due to the cognitive demands related to interpreting the feedback but also generating commands using pattern classification. Nevertheless, it is promising that the participant could successfully exploit the closed-loop control of multiple DoFs after only brief training. Prolonged training is likely to have a strong effect, particularly on time efficiency. Finally, although the participants controlled the DoFs sequentially via pattern classification, the same feedback scheme can be used in combination with regression to allow simultaneous control. In this case, the elicited tactile sensation changes both in the location and extent of spreading concomitantly. How well and how fast the participants can accommodate such dynamically changing feedback remains to be tested.

Compact and conventional feedback lead to same level of performance and user acceptance

Our initial assumption was that the participants would perform better with the Conventional vibrotactile feedback scheme with respect to Compact vibrotactile feedback scheme, as the interpretation was simple in this case (identifying a single active motor). The results demonstrated that the two schemes performed similarly and that the vibrational feedback is a good strategy to provide proprioceptive information. It guaranteed statistically better performance with respect to no-feedback condition.

In addition to similar accuracy, the subjective preference was similarly distributed between the two vibrational encoding schemes (6 vs. 4 in able-bodied and 2 vs. 2 in amputee participants). For able-bodied participants, the preference was slightly in favor of the Conventional feedback scheme. Nevertheless, our results demonstrate that the Compact solution can still be used in cases where there is not enough space to fit all vibromotors. In some participants, this could decrease the user experience (lower preference) but will not significantly jeopardize the performance. However, such preferences were expressed after a single session of training and it is possible that longer training change participants’ choice.

The results for the efficiency complement those for the accuracy. Without feedback, the control was challenging and so the participants were most efficient (straight and quick) but least accurate. In all feedback conditions (except for one case), the efficiency was not significantly different. Visual feedback was again superior as it resulted in the highest accuracy without prolonging the generated trajectory or trial time (even showing a trend towards slightly more efficient performance).

Implications for prosthetic development

The results obtained in the present study are important because they show that the number of vibromotors can be safely decreased from twelve to eight, without loss of performance. Therefore, excess vibromotors can be removed to make the interface easier to integrate, or they can be used to encode additional information, such as wrist flexion/extension or grasping force. Such an interface could be used to provide simultaneous feedback of three different DoFs: hand aperture, wrist rotation, and wrist flexion and extension, and hence, the full kinematic state of a prosthetic hand, to get complete proprioceptive feedback.

Apart from the primary objective of comparing the two vibrotactile approaches, the study also evaluated the novel method in comparison to the incidental feedback, and the results obtained were similarly promising. The improved performance in comparison to the absence of feedback indicates that the participants effectively utilized vibrotactile stimulation during the task. This suggests that the stimulation was advantageous, as relying solely on feedforward control and muscle proprioception did not enable them to perform the task equally well. When compared to incidental feedback, vibrotactile stimulation did not provide additional benefits. In fact, visual feedback was the most effective overall and this was an expected outcome because the participants could clearly see the prosthesis on the screen and align it precisely into the target configuration. Otherwise, the similar performance of vibrotactile and auditory feedback was somewhat unexpected since the latter did not convey the prosthesis position explicitly. Nevertheless, the participants could hear when the prosthesis started moving and estimate the movement speed [55], then use this information to predict the position. This outcome underlines the power of incidental feedback that has been shown in other studies to be valuable for expert users [56].

While this very effectively demonstrates the challenges of sensory substitution, it does not invalidate the potential clinical utility of the vibrotactile feedback, as explained below. Visual feedback might be very effective, but it requires constant visual focus on the prosthesis, and such overreliance on vision is indeed a well-known behavior of prosthesis users [57]. The provision of vibrotactile feedback might motivate the users to explore other strategies, as it would allow adjusting the prosthesis configuration without necessarily looking into it (as demonstrated in the present experiment). This can be additionally facilitated by the fact that the interpretation of such feedback is likely to become automatized over time. Of course, this is highly speculative, and needs to be tested in future work. Similarly, the auditory feedback facilitated the understanding of the prosthetic motion, however, the noise from the prosthesis motors is an undesirable feature and the quality of this feedback source is likely to be prosthesis dependent (i.e., a more or less noisy prosthesis). Future prosthesis development will focus on noiseless motors, which will eventually eliminate this source of feedback.

The setup for closed-loop control used in the present study was realized using an embedded platform that is ready to be integrated into a prosthesis for rapid clinical use. The embedded platform includes recording of EMG signals, pattern classification, and control of vibromotors, as well as bidirectional communication with the real prosthesis. In general, tactile feedback has rarely been demonstrated in combination with pattern classification and multiple-DoF control [35], especially when using an embedded setup [58].

Recently, multichannel electrotactile stimulation was used to convey two degrees of freedom [35]. The authors exploited the compactness of electrotactile stimulation to provide feedback through a flexible electrode integrating an array of 16 pads. Nevertheless, the encoding scheme utilized a Conventional approach in which the array was segmented into non-overlapping sectors allocated to different DoFs, which is equivalent to the multiarray feedback of the present study. An additional advantage of the vibrotactile interface is that it does not interfere with the recording of EMG, and hence does not require specialized hardware and software.

Limitations

Testing all conditions in the same pool of subjects would be the best approach. However, doing this in a single session would be excessively long, and mental fatigue could have affected the results. Due to this and logistics reasons, the test and control conditions were performed by two separate groups of participants. Importantly, this was considered when statistically comparing the conditions, as explained in the Data analysis section.

Future work

Following the encouraging results of the present study, especially regarding amputee participants, the next step in our research will be to integrate the system into a prosthesis socket and test closed-loop control during functional prosthesis use. A direct mechanical connection between the body and the prosthesis will provide other sources of incidental feedback that could not be explored in the present study, for instance, motor vibration which could be especially prominent when using osseointegrated prosthetics. This would be the ultimate test assessing the benefits of vibrotactile stimulation when all the other feedback cues are simultaneously available. Additionally, we expect that the proposed vibration feedback scheme, in a chronic or daily usage, could be integrated in the mental scheme, giving the possibility of unconscious automatization. This aspect will be investigated in future work during longer at home experiment.

The final aim of our work is to accommodate all DoFs of the Hannes prosthesis for both control (pattern classification) and feedback (full proprioceptive and exteroceptive substitution). Importantly, the proposed compact approach is general and can be used to convey different information, e.g., not only position, as in the present study, but also grip stability [59], EMG biofeedback [60] or prosthesis actuation speed [61], individually or in combination.

Conclusion

The present study proposes a novel compact approach to provide artificial vibrotactile feedback for two DoFs using a single array of vibromotors. In the proposed approach, wrist rotation was conveyed by moving the peak of sensation around the forearm, while the hand aperture was transmitted by increasing the spread of sensations from the peak location as the hand was opening. Hence, the two variables were conveyed using intuitive mapping. The tests demonstrated that the novel approach performs similarly to the conventional feedback scheme, in which separate arrays are used to convey individual DoFs. The present study therefore shows that the number of vibration motors required to provide feedback can be decreased without any significant performance loss. Furthermore, the novel interface was compared to visual and auditory feedback, as well as no-feedback condition and the results confirmed that the vibrotactile stimulation was indeed useful for control. This, in combination with the fact that feedback was integrated with pattern recognition, represents an important step towards the development of clinical applications that embed both feedback and control into a single prosthesis socket.