Keywords

1 Introduction

Sensory feedback is essential for dexterous manipulation and its absence in hand prostheses hinders the correct achievement of numerous fine-motor tasks, such as grasping, stroking or throwing [1]. Human sensory dexterity mechanisms are inherently dependent on both complex nervous arrangements and two sensory systems: namely proprioception, which tracks the motion of our hand and its muscular effort; and touch, which communicates key information on environment interactions [2]. With visual guidance being an imperfect surrogate for its highly intricate somatosensory complement and the fact that our motor systems are inherently coupled with our dexterity [3, 4] the underlying necessity of a sensory feedback system for prosthetic hands is clear.

The vast majority of works in the literature has focused on creating haptic feedback systems to interpret different aspects of touch, such as tactile sensations or contact forces. To achieve this, the prosthetic hand used is typically equipped with some form of touch sensor [5]. However, the sensing capabilities of prosthetic hands are currently still limited, as the underlying technology lacks the sophistication of the natural counterpart. Nonetheless, vibration and pressure-based feedback have shown promising results in rendering touch, to the point where individual finger contacts can be discerned by users [5,6,7].

When rendering proprioception, both the position and force applied by the hand must be encoded into the feedback applied to the user. This is considerably more applicable to hand prostheses, given that the position and applied force of each controlling motor can be monitored without any specialised sensing technology. Despite this, very little work exists reproducing elements of proprioception to prosthetic hand users. Of note, [8] utilised a rolling mechanism to encode the position of the whole hand, resulting in users being able to identify small changes in grasped object size.

Regarding modes of haptic feedback for prosthetic hands, previous works have mostly focused on a single form of information [9], thus inherently limiting the level of perception available to the user—often aiming to represent the state of the entire hand [10, 11], rather than doing so at a finger level. An ideal haptic feedback system should indeed utilise mutli-modal feedback to encode different aspects of proprioception and touch. Multiple feedback sites should also be utilised to allow the user to differentiate feedback from individual fingers.

Fig. 1.
figure 1

Overview of the introduced multi-modal haptic armband. Signals are read from a prosthetic hand and translated to vibration, tangential position, and normal position feedback.

Full size image

The aim of this work is to develop a multi-modal haptic feedback armband, shown in Fig. 1, which is able to encode key aspects of somatotopic proprioception and touch, suitable for use with prosthetic hand technology not having specialised sensors. In particular, three feedback mechanisms are considered: vibration, normal displacement, and tangential position. Our device stimulates different corpuscules by implementing vibrations (Pacini), dynamic skin deformation (Meissner) and directional shear forces (Ruffini) [12]. Our findings indicate that combining the vibration and normal displacement mechanisms in the armband enables the rendering of sensory information involved in finger contact to a higher level than vibration alone. Tangential motion alone enables users to identify a limited but promising amount of finger-level motion. Results of user evaluation of the armband also show important differences in sensitivity to feedback of locations around the upper arm, which should be taken into consideration in future research and development.

2 Multi-modal Haptic Armband

2.1 Rendering of Feedback

The multi-modal haptic armband is used to render feedback from a prosthetic hand using the position, \(\theta _m\) measured by the encoders, and torque, \(\tau _m\), of the motors driving each finger. In this work, we specifically consider the OLYMPIC hand [13], in which each finger is driven in the flexion direction by an individual motor, while springs mounted on the dorsal side of each finger drive extension. Contact can be detected as the hand closes around an object, and relayed to the user using the feedback mechanisms of the armband; vibration for indicating touch sensations (from motor position and torque), normal displacement for rendering contact force and force elements of proprioception (from motor torque), and tangential position for rendering position elements of proprioception (from motor position).

Vibration. Vibration is used to render the tactile element of contact. Without a direct sensor mounted on each fingertip, contact must instead be inferred from motor position and torque. This can be achieved by monitoring the acceleration of each finger motor; large deceleration indicates that the finger is making contact with an object. The vibration of finger i is applied as a pulse of fixed duration \(T_v\), and intensity \(k_v\), upon acceleration falling below a negative threshold, \(\ddot{\theta }_v\):

$$\begin{aligned} v_{i}(t) = {\left\{ \begin{array}{ll} k_v &{} \ddot{\theta }_{m,\,i}< \ddot{\theta }_v\;\;\;\;\;\;\;\;\;\text {for } 0< t < T_v \\ 0 &{} \text {Otherwise} \end{array}\right. }. \end{aligned}$$
(1)

Normal Displacement. When each finger is stationary, the motor torque required to hold the finger at its current position is equal to the combined torque of the extension springs acting on each joint [14]. A linear estimation of contact-free motor torque, \(\hat{\tau }_{s,\,i}\), can therefore be calculated. Motor torque is therefore approximately linear to position when each finger of the hand is contact-free. Assuming the skin behaves according to a visco-elastic model, the contact is stable, and displacement is small, then contact force can be rendered by applying a proportional displacement normal to the surface of the skin. The displacement applied to the user is proportional to the difference in current motor torque and the contact-free stationary torque, scaled by \(k_y\):

$$\begin{aligned} y_{i} = k_y(\tau _{m,\,i} - \hat{\tau }_{s,\,i}). \end{aligned}$$
(2)

Tangential Position. Proprioceptive position feedback is achieved by relaying the current motor position to the position of a linear drive, \(x_i\), moving tangential to the surface of the skin, with linear scaling \(k_x\):

$$\begin{aligned} x_{i} = k_x\theta _{m,\,i}. \end{aligned}$$
(3)

2.2 Mechanisms of Feedback

The multi-modal haptic armband is composed of five modules equally spaced around the arm, each corresponding to one of the user’s fingers. Each module, shown in Fig. 2, consists of three feedback mechanisms: a vibration motor rendering tactile contact sensations, a servo-motor to apply displacement normal to the skin expressing contact force and force elements of proprioception, and a linear drive that moves tangentially to the skin rendering position elements of proprioception. Each individual module is self-contained and has a footprint of \(68\times 40\) mm, a height of 33 mm, and a mass of 56 g, which makes the armband suitable for daily use due to its relatively compact size and small weight. Studies have shown that the upper-arm region in which our research is being implemented has a sensitivity threshold distance of around 30 mm [15] which validates the 40 mm width of the design.

Fig. 2.
figure 2

(a) User wearing the armband, modules labelled. (b) Schematic of the feedback mechanisms.

Full size image

The vibration motor is placed in the upper inside wall of the casing such that vibrations are distributed throughout its shell, as to render non-position-specific feedback. The servo is mounted on a carriage attached to both the linear drive and two support rods allowing for smooth travel and the application of normal displacement at a desired position. To keep module size to a minimum, a small 10 mm diameter stepper motor was used, coupled with a 38 mm threaded rod to form a linear drive. An extension to the servo motor (Tower Pro, SG90) horn was also designed to create better contact with the user’s skin using a spherical tip.

Fig. 3.
figure 3

(a) Sensory combinations with a duration of 2.0 s used in finger differentiation experiments; vibration alone, and both vibration and normal displacement. (b) Summary of finger differentiation experimental setup and performance metrics calculated. For any trial, each module has a probability of 0.5 that it will apply feedback.

Full size image

3 Experimental Evaluation

In order to evaluate the haptic armband, 5 users wore it on their right arm, with the first module aligned with the centre of the bicep, and subsequent modules located around the upper arm ordered in the direction of the outside of the arm. By design, the armband blocks the user’s view of each feedback mechanism, and headphones were used to block audio cues.

Just-Noticeable Difference. The just-noticeable difference (JND) of the servo-based force feedback applying a nominal normal displacement of 3.0 mm and the linear position feedback at a nominal tangential position of 17.5 mm were individually evaluated. One feedback module was used to deliver paired stimuli, using a staircase procedure to find the JND. This was repeated for each module of the armband, evaluating the sensitivity to feedback of 5 points around the upper arm. JND is important in this case as it allows us to gauge how small changes in stimuli are perceived. When grasping fragile or delicate objects, slight variations in position and force can be of great consequence, so understanding the perceptual limits of these feedback mechanisms is critical.

Finger Differentiation. These experiments were performed to evaluate the ability of the haptic armband to provide finger-level feedback that is differentiable to the user. First, using a digital twin of the OLYMPIC hand [14], users were asked to identify which fingers of the prosthetic hand made contact with a virtual object based on vibration feedback alone, then the experiment was repeated with normal displacement feedback also present. At each trial, each finger randomly contacted the virtual object with a probability of 0.5. In both scenarios, 100 trials were used and feedback was applied for a total of 2.0 s; for vibration and normal displacement, vibration was applied for 0.75 s, and a normal displacement of 3.0 mm applied for 1.75 s, such that the two feedback mechanisms overlapped in application Fig. 3(a).

Two metrics are calculated from users results; individual finger accuracy and exact combination accuracy. As summarised in Fig. 3(b), individual finger accuracy allows us to monitor how successful the user is at detecting contact on each finger. Exact combination accuracy is a harsher metric, only counting when the exact combination of fingers has been correctly identified. This may be a more appropriate metric when considering prosthetic hands; users should be able to identify precisely which fingers are making contact with objects if they are to achieve dexterous control of the prosthesis.

Fig. 4.
figure 4

Left: JND of normal displacement for each module, measured with a nominal displacement of 3.0 mm. Middle: JND of tangential position for each module, measured with a nominal position of 17.5 mm. Right: PSE of tangential position for five modules used simultaneously (note that module 4 suffered an electrical fault and only 2 user results are presented).

Full size image

Point of Subjective Equality. Measuring the point of subjective equality (PSE) of position feedback is crucial. If this armband is to be used during prosthetic hand control, the user must be able to control their hand to a desired pose and be confident that, based on the position feedback they receive, the hand has indeed reached that pose. In this experiment, users received position feedback from a control pose, then were asked to adjust 5 sliders on a user interface in 0.24 mm increments until they produced a position feedback perceived as equal to that of the control pose.

4 Results and Discussion

To measure the efficacy of the system, JND was measured for normal displacement and tangential position feedback, which measures the amount of change in a stimulus that allows it to be detectable at least \(75\%\) of the time, as estimated by fitting a psychometric curve to user responses. As seen in Fig. 4(left and middle), JND of modules located at the rear of the upper arm (modules 3, 4) was significantly lower than modules at the front of the upper arm (modules 1, 2, 5), with median values of 0.72 mm versus 0.45 mm for normal displacement, and 3.06 mm versus 2.53 mm for tangential position. This is expected, considering the high sensitivity of glabrous skin [2], which is located primarily on the anterior and medial regions of the upper arm. The difference in JND around the upper arm can be used in future to inform module placement and specific feedback mapping that takes this into account when rendering sensory cues.

Finger differentiation results reveal that multi-modal feedback generally improves user ability to identify whether a finger has made contact (see Table 1a), although user performance with vibration alone is still high. The ability of users to recognise the exact combination of fingers in contact at any point is heavily dependent on the number of fingers in contact (see Table 1b). Combining vibration with normal displacement feedback improves exact combination accuracy considerably. With vibration alone, exact combination accuracy when four fingers are in contact is little over random chance (\(1/32 = 0.031\)), at 0.048, whereas when normal displacement is included, this rises to 0.130. Although each module is larger than the sensitivity threshold distance of the upper arm (30 mm), vibration transmits through the skin and can ‘blur’ feedback [6], which may account for the poor finger differentiation performance of vibration alone.

Table 1. Finger differentiation accuracy for users identifying contact of digital twin fingers using vibration (V) and normal displacement (ND) feedback. (a) Individual finger accuracy. (b) Exact combination accuracy.
Full size table

Point of subjective equality results, shown in Fig. 4(right), reveal that user ability to interpret tangential position is accurate to within \({\pm }20\%\) when multiple modules are active. Unfortunately, module 4 suffered electronic failure during evaluation, so results for it are only present for two users. PSE for remaining modules follows a similar pattern to previous results; the anterior region of the forearm is more sensitive to stimuli, which may account for user over-estimation of tangential position in modules located in these regions, while the reverse is true for modules located toward the less sensitive posterior regions. It is hypothesised that PSE results could be improved by increasing the travel of the linear drive, and improving the geometry of the servo arm to contact more skin.

5 Conclusion and Future Work

In this work, we have presented a haptic feedback armband that is capable of rendering sensory aspects of touch and proprioception of a prosthetic hand. User evaluation revealed that JND of normal displacement feedback and tangential position feedback is highest on the anterior regions of the upper arm, dropping significantly in non-glabrous regions at the posterior of the upper arm. Using normal displacement to translate contact force supplements vibration-rendered touch sensation and greatly improves user ability to identify exact fingers in contact with objects. Users were able to resolve tangential position to within a noticeable difference as little as 2.0 mm for a single finger, but PSE results indicate that improvements are required to relay accurate position information to the user. Alternative designs investigating a larger range of motion, improved reliability of the linear drive, and different roller geometries should be considered.