Examples of Haptic System Development

Abstract

In this section, several examples of task-specific haptic systems are given. They shall give an insight in the process of defining haptic interactions for a given purpose and illustrate the development and evaluation process outlined in this book so far. Examples were chosen by the editors to cover different basic system structures.

Section 14.1—User Interface for Automotive Applications presents the development of a haptic interface for a new kind of user interaction in a car. It incorporates touch input and is able to simulate different key characteristics for intuitive haptic feedback. Sect. 14.2—HapCath describes a comanipulation system to provide additional haptic feedback in cardiovascular interventions. The feedback is intended to reduce exposure for both patient and physician and to permit new kinds of diagnosis during an intervention. Sect. 14.3—FingHap—Haptic Finger Rehabilitation Device presents a finger rehabilitation system with feedback on fingers. It moves the fingers in their normal trajectory and according to the improvement of patient, it changes the helping torque.

1 Touch Input Devices

Jörg Reisinger

Mercedes-Benz AG, Sindelfingen, Germany

{joerg.reisinger@mercedes-benz.com}

Ingo Zoller, Peter Lotz

Continental Automotive GmbH, Babenhausen, Germany

{ingo.zoller@continental-corporation.com; peter.lotz@continental-corporation.com}

Dongkill Yu

LG Electronics

{dongkill.yu@lge.com}

Since the last issue, touch input devices have taken control of the vehicle interior. Familiar button panels have almost disappeared from the vehicle, reduced to individual buttons. Rotary push controls, which have taken over the leading role in the 2000s advanced interaction concepts, have turned out to be an interim solution, typically offered as a second solution for traditional customers in between.

The change to touch input devices promises an unexpected flexibility, no matter whether touchpad or touchscreen, content is no longer static and especially intuitive to use on a touchscreen.

As already achieved with e.g. central control elements, the advantages of a drastic reduction in functions from many to just a few elements, are in the clear structured integration of the many new functions, as well as cost. Further cost advantages arise when reducing from a separated display and input device to a touchscreen. Actually an ideal solution; however, a major disadvantage is a complete visual focus by eliminating any other feedback. Acoustic noises as well as the haptic feeling of the control element are not available to the customer anymore.

Out of a technological view an acoustic feedback generated by perhaps already existing or additional audio systems is a comparably simple and already known possibility. Instead, adding haptic feedback means the integration of additional, completely new technologies into the overall system with many new and varying boundary conditions. It is therefore not surprising that due to the cost and availability of ready-to-use systems, high quality active haptic feedback currently only is available in the exclusive segment of (automotive) touch input devices.

This chapter provides an overview and an insight into the technological differences of touch-operated haptic devices in the premium automotive segment. The systems currently in use are all working with vibro-tactile technologies, i.e. the surface vibrating in the haptically perceivable frequency range.

The technical framework conditions and changing requirements - primarily driven by the design and size of the components—result in different concepts that manifest in a significant technological progress.

Man is a multimodal perceiving being, so that apart from haptics, considering topics like design as a visual aspect and noise an auditory aspect. Technical possibilities and limits must not be underestimated.

The previous edition of the book describes the haptic touchpad introduced into the 2013th C-Class in detail (Fig. 14.1) as well as following ones. Since then, there has been a strong evolution. While haptic touchpads dominated at that time, Audi, followed by the Porsche Tycan, introduced the first haptic touch screens in 2017 as a sidekick concept. A new haptic feedback concept introduced in 2020 in the Mercedes-Benz S-Class (Fig. 14.2) and its all-electric brother EQS achieves new sizes and qualities. We will take insight into this application as well. Please consider, the whole chapter results out of experiences during product development, containing many thoughts, not always scientifically worked out in detail.

Fig. 14.1

Applied automotive haptic device—touchpad in the Mercedes-Benz C-Class, Year 2014

Fig. 14.2

Haptic Touchscreen of the 2020th Mercedes-Benz S-Class

1.1 Direction of the Stimulation

In the 2000s, several studies conducted on rapidly adapting (RA) mechanoreceptors. While for example in 2005 [3] dealt with the frequency response of the Pacinian Corpuscles, in 2010 [11] showed that 3-dimensional stimuli can be reduced to a single vibrational direction without significant loss of information. Finally, it showed that haptic feedback does not have to take place in the normal direction of the skin surface of a finger, but it can take place in every direction. [28] presented a haptic feedback device for haptic clicks by lateral movements of the surface. In general, this opened up new approaches to design haptical devices like touchpads or displays by further technical possibilities.

1.1.1 Sidekick Versus Normal Actuation

As a result, due to certain technical advantages of this principle, first products appeared in the market. However, somehow a tendency achieved telling that the sidekick concept might be “mandatory” for haptic feedback, what is quite doubtful. To be clear, this new opportunity is offering a new technical approach of how the surface can be stimulated without a negatively influenced perception, but it is not the only valid solution. In any case, however this insight helped the breakthrough of haptic feedback in the automobile very much; it maybe made it possible.

Overall, there are six spatial directions suitable for technically creating a vibrotactile haptic feedback. Two of them are in normal direction to the surface, four of them in lateral direction, as indicated in Fig. 14.3; we will now have a closer look at advantages and disadvantages of each.

Fig. 14.3

Principles of lateral and normal displacement of the surface to stimulate haptic feedback

1.1.2 Advantages of the Sidekick

The lateral movement of the surface as sketched in Fig. 14.4 offers several advantages. Mainly uniform haptic feedback due to the high rigidity of the surface plate in the excitation direction helps to move each surface point nearby the same, while activating almost no surface vibration modes. This helps uniformity from a haptic point of view as well as acoustics.

Furthermore, separating the actuator’s acceleration direction and the user’s operating direction. This has the advantage of minimally affecting the force measurement by the haptic impulse improving the quality and reliability of the force signal during a haptic event. In terms of installation space, too, the concept initially offers advantages. For example, there is no need to distribute actuators across the surface or to build complex mechanisms to generate a uniform behavior there. At all this is limited to overall size of the actuated area, because increasing the area, requires further structures to get the surface stiff enough to transfer the impulse (Fig. 14.4).

Fig. 14.4

Principle of the “Sidekick” lateral haptical stimulation

1.1.3 Sidekicks’ Disadvantage

Overall, this principle seems to be a perfect solution—and already works very well in several applications—but there are also disadvantages we need to consider as well.

Depending on the product design, the gap surrounding the active area requires additional space for the lateral movement of the surface next to other technically required tolerances. Consequently, it has a direct influence on the design concept of the product, can contradict its philosophy and requires creative workarounds.

The transfer of forces from the surface to the skin via shear forces leads to another point: The vibration motions require friction between finger and surface to transfer them to the fingertip. Thus, a surface with a low friction coefficient can reduce perception of haptic stimulus. Low-friction contact surfaces are therefore counterproductive with regard to the haptic intensity, which in the case of surface-normal vibration is of minor importance due to the form locking of the effects direction.

What is initially advantageous in the technical design can become a disadvantage when multi-finger-feedback is required. For example, when several users operate the device simultaneously. However, the movement of the entire surface is yet another technical aspect. The system must always move the entire mass of the surface.

The mass increases disproportionately, because as the size increases, more and more measures have to be taken to maintain the moment of inertia, which leads to a large increase in mass.

The Actio-et-Reactio principle (Newton’s third law) inevitably leads to another topic: Generating the impulse on the surface (= Actio), is inducing the Reactio in the housing of the component and beyond into the dashboard. It is easy to imagine that this impulse at the dashboard, comparable to the body of a musical instrument, produces strong acoustic reinforcements that can differ greatly from the general quality expectations. To reduce the impulse partially and liquidate mechanically, so-called “tilgers” or “absorbers” are moving against the initial acceleration. Unfortunately, an absorber additionally increases the moving mass. So that the size or mass of the active surface is limiting this approach as well. Fair to say, this effect is also occurring in systems accelerated in normal direction as long as they are using quasi-static surface vibrations, i.e. the entire surface designed moving uniformly up and down.

1.1.4 Sidekick Application

Small lightweight surfaces such as the Audi Touchpads or in non-automotive market the trackpads inside the Apple MacBook since 2015 ([6]) work really well. At all, increasing size and mass limits the areas of application. Audi [26], and Porsche Displays [5] as in Fig. 14.5, both use the Sidekick principle and accelerate a mass of several kilograms. The division into two separate displays may be because of this reason.

Fig. 14.5

Audis’ Touchscreen introduced in the A8 2017, using a sidekick haptic feedback [26]

1.2 Lateral Differences of Excitation Directions

The lateral deflection of the surface can occur as shown in Fig. 14.6 in different directions regarding the finger posture. The mechanical impedance of the human finger varies noticeably with respect to the load direction, so we describe a few observations and thoughts in the following.

While the direction of pulse to the right or left has a comparable behavior due to an almost symmetrical finger impedance in those directions, the forward and backward direction strongly differ. A rocker switch, as shown in Fig. 14.7 can demonstrate the influence of forward and backward. When operating both directions, “forward” and “backward” feel significantly different. In principle, the reason can lie in the component itself. However, if the component is turned and both directions are changed, the previously experienced perception does not change and remains oriented in the finger direction and there seems no component alignment. Obviously, a difference in directional finger stiffness for push and pull is responsible for that difference, that why assuming that the finger direction is influencing the perception of the effect. Therefore, in the following we are discussing how the direction of active surface excitation may be influencing the coupled system’s vibrational properties.

Fig. 14.6

Potential lateral excitation directions of the surface

Fig. 14.7

Dynamic switch C-Class

1.2.1 Coupling Between Finger and Surface

The normal movement of the C-Class (205) touchpads’ surface introduced in 2013 moves the surface “downwards” or “away” from the finger Fig. 14.8. Due to a slight moving surface and the primary focus on push haptic feedback, it behaves like a typical micro switch with a snap disk underneath. The surface jumps down, when reaching a certain force level. There are two relevant acceleration maxima of the vibrational event, the “breaking away” and the “hitting the ground”, influencing the haptic intensity of the feedback. Typically, the customers did not identify it as an artificial haptic system. Furthermore, they qualitatively perceive a mechanical micro-switch-based feedback. The “Wow” is great when customers realize this, for example, deactivating the snap i.e. push feedback when placing the palm on the touch device when operating the rotary controller.

Fig. 14.8

Movement “Downward”/“away from” the finger. The initial peak will decouple from the finger, the following impacting peak will couple into the finger

Looking into technical details, Fig. 14.9 shows the solenoid actuator of the floating touchpad of the C-Class 2013. Its slim shape is perfect and predestined for the Cobra-like touchpad floating above the rotary controller. Integrating the coil in the PCB, makes it flat across the whole floating area and the spread actuator is for pulling the steel actuator downward uniformly to prevent inhomogeneous behavior. A parallelogram solid-state guidance supporting the vertical movement of the actuator without tilting towards the PCB of course keeping the position laterally. An optical sensor detecting the displacement of the actuator used for sensing the force.

Fig. 14.9

Basic actor design—PCB between two steel plates. The springs represents the elastic component of the module, at the same time they provide the parallel guiding system. It pulls the surface away from the finger

An increasing mass and introducing search haptic features, i.e. a feedback appearing when sliding the finger laterally across the surface; it may be worth considering also the direction of actuation due to efficiency of the effect transfer to the finger. The direction “against” as shown in Fig. 14.10 may help to improve the mechanical coupling by initially moving the surface against the finger, increasing the coupling, supporting the first impulse. As an example, the 2018th A-Class Touchpad using a solenoid actuator working “against” the finger is following.

Fig. 14.10

Movement of the surface “against” the finger. The initial pulse is directing the first pulse against the finger creating a maximum coupling

1.3 Increasing the Mass—Touchpad as a Sculptural Element

New design requirements and UI concepts changed the boundary conditions of the previous haptic concept, which was presented as a sculptural touch-only touchpad in the 2018th A-Class, as shown in Fig. 14.11.

Fig. 14.11

The 2018th A-Class’ Touchpad is “kicking against” the finger

1.3.1 Frequency Behavior

In contrast to the touchpads introduced so far, the new generation of touchpads move the entire sculpture to a greatly increasing moving mass, having a 10 times higher mass compared to the floating touchpad or other touchpads in market.

The influence of mass and stiffness on natural frequency in Eq. 14.1 shows that increasing mass decreases the natural frequency of the system accordingly, in consequence, frequencies above the natural frequency can no longer be stimulated as efficiently. The frequency-dependent sensitivity of the mechanoreceptors (Sect. 2.1.1) requires a stimulus in a specific frequency range when designing the system.

$$\begin{aligned} \omega = \sqrt{\frac{c}{m}} \end{aligned}$$

(14.1)

$\omega$: natural frequency, c: stiffness, m: mass

Increasing the system stiffness is one option compensating the increasing mass, keeping the natural frequency that in turn has further effects: First, compare IDM in Chap. 13, reducing the influence/variance of human finger impedance as the coupled impedance of the control becomes stiffer and thus gains dominance (compare Eq. 14.2). However, in principle this makes the system more robust against variation of finger impedance.

$$\begin{aligned} c_\textrm{coupled} = \frac{c_1\cdot c_2}{c_1+c_2} \end{aligned}$$

(14.2)

$c_\textrm{coupled}$: coupled stiffness, $c_1$, $c_2$: single components’ stiffness

In addition, compare Eq. 14.3, both the higher mass as well as the stiffness increase the required energy to stimulate the surface haptically correct.

$$\begin{aligned} w = \frac{1}{2} m \omega ^2 A^2 = \frac{1}{2} c A^2 \end{aligned}$$

(14.3)

Total energy of a vibrating mechanical system, with W: Energy, A: oscillation amplitude, $\omega$: natural frequency, c: stiffness, m: mass.

Therefore, the actuator generally needs more power while maintaining the momentum to drive the current frequency range, as you can see in the simulation results in Fig. 14.12. It shows the frequency domain of the coupled system consisting of control element and finger, in the example changing the impedance of the finger. The amplitude of the vibrations around 400Hz is strongly reducing due the increasing impedance. That why the energy in this frequency range needs to be increased to reach the same level. However, this higher energy now must be controlled and minimized in order to avoid negative effects.

Fig. 14.12

Influence of the finger impedance on the systems frequency behavior. I. Standard finger impedance, II. Damping x 3, stiffness x 10, III. Mass x 10 [29]

1.3.2 Reduction of System Noise

The mechanical haptic impulse produced of course follows Newton’s third law of interaction, actio et reactio. This means that the haptic impulse in the surface leads to a more or less powerful impulse in the environment, which means the center console or dashboard in the vehicle. Amplified there like in a guitar body it may lead to disturbing noises that are unacceptable with the need of reduction and compensation. However, a simple reduction of intensity is counterproductive, as haptic feedback can become too weak for interaction. At all, there are approaches reducing negative effects of this disturbing external impulse, when not being able excluding it fundamentally by construction, shortly discussed in the following.

1.3.3 Increasing the Signal Vibration Duration

The simplest method for energy minimization may be a reduction in amplitude by a longer continuous vibration due to a lower damping time constant. This is actually a regression to a vibrating effect, as we know it from the earlier mobile phones. This effect does not seem crisp and high quality, but rather like the buzzing of a bee, which rather has the character of a warning signal than a pleasant and valuable feedback. [23] describes this difference and Fig. 14.13 showing a strongly dampened short, crisp effect compared to the vibration-like effect.

Fig. 14.13

comparison of short and increased vibrotactile haptic feedback

At all, the disadvantage of haptic quality and haptic “meaning” has its advantage in low cost components, small size and a low short-term mechanical energy request, by stretching the energy consumption in time, reducing the magnitude required. At all, it does not guarantee having no noises in the environment as well. An active haptic system with an eccentric rotating mass motor (ERM) currently used in the VW ID3 steering wheel switch.

1.3.4 Mechanical Impulse compensation

To reduce the environmental impulse, mechanical solutions for its liquidation can be used (so-called absorbers). A combination of mechanical frequency filters, consisting of specific impedances and a moving mass working against the moving surface in the background of the system can help to reduce the reactive impulse. However, this leads to an additional increase in mass, which lowers natural frequency and leads to additional energy requirements. Therefore, implementation needs a very thorough theoretical design.

1.3.5 Leaving Out of Disturbing Frequencies

Inspired by the thoughts of frequency- or input-shaping as described by [7], a simplified approach using a Fourier series as a sum of dampened sine wave signals of different frequencies and damping properties. This Fourier series deliberately does not add the noise producing, disturbing resonance frequencies of the depending resonance bodies as well as their (sub-) harmonic frequencies.

More efficient but also more difficult, seems the frequency-input shaping approach itself, as described by [7]. This optimized approach regarding energy efficiency due to using the full signal bandwidth and only removing the interfering frequencies. Both approaches thus derive the fundamentally same principle by suppressing the disturbing noise frequencies. At all, it is unfavorable if the interfering frequencies come very close to the frequencies required for the good haptic stimulation of the mechanoreceptors as described in (Sect. 2.1.1). Removing frequencies in these frequency ranges is reducing the energy and thus the efficiency of the stimulus. In general, it is an approach to optimize the feed forward signal by frequency selective signal design.

1.3.6 Active Damping by Frequency Increase

As so often, there is also a conflict between energy efficiency and quality here. As mentioned before, a high-quality haptic click signal has a very short duration. An increased mechanical damping can ensure this, but requires higher performance to obtain the same intensity of a signal.

One common damping method is using an opposing impulse derived out of measuring the signals’ response. This requires a measurement for calibrating the system’s parameters, mainly the delay of activating the opposing impulse. Distortions of the system like drift or thermal deviations appear counterproductive but typically are not too significant. An alternative principle described in [20], uses the previously described Fourier series approach using frequency shift. Therefore, increasing each dampened frequency individually to dampen the energy of that specific frequency. The frequency increase in turn effectively dampens the overall signal, realizing energy-efficient mechanics with a low mechanical damping, as applied in the Touchpad generation of the 2018 A-Class.

1.3.7 Actuator Integration and Handheld Versus Mounted Devices

In the case of installed haptic devices such as touchpads or automotive devices, the integration of the actuator typically realized by clamping it between the housing and the actuated surface. Mobile devices, on the other hand, typically use single side mounted, not clamped actuators, accelerating a mass as a counter weight for the generation of the mechanical impulse. However, except for a few exceptions and against all expectations, typically the actuators of mobiles mounted on the housing side instead of the display side, where the perception generally is expected to take place. While the Linear Resonant Motors (LRM) is increasingly interspersed with high-quality haptics, with Apple’s Haptic Engine$\text{IMAGE}$ as the most popular one. Very first mobile devices were using Eccentric Rotating Motors (ERM) generating a vibrating-type feedback rather than the precise click. To reduce the periodically appearing clicks to a single one, an extra effort is inevitable. Common for both described systems is that the haptic impulse is transferring into the housing of the device. So far, no significant effect is perceptible on the display itself in a mobile device, but only transmitting via the casing, held in the second hand. This simply testing out by placing the device on a table and operating it there, the haptic feedback may not be perceivable. Concluding, Haptics transmitted via the housing is integrating into overall perception via the second hand.

One exception is Google’s Pixel 5, introduced in November 2020. Installing the actuator directly on the backside of the display’s surface directly stimulating the resonant modes of the display’s surface.

The permanent magnet of the LRA is the moving part, working as a counter mass for the generation of the mechanical impulse, but working in normal surface direction. Additionally it supports the devices’ Audio.

The fact that primarily causing system’s noise by the vibrations transmitted into the housing, it is of significant advantage inducing the surface’ stimulus without inducing into the housing. This allows driving the actuators more efficiently and stronger, due to the reduced noise coupling. This finally refuses the argument that the installed clamped systems’ actuators may offer more “impulse” due to its stiff setup. However, acoustic noise is limiting and reducing the “useful impulse”.

1.3.8 Solenoid Versus Voicecoil

Solenoid actuators are comparably cheap and can generate high dynamic forces. Unfortunately, the unidirectional properties is limiting the possibilities strongly, i.e. applying negative forces is not possible except with a mechanical spring. This nonlinearity greatly limits signal- and control-related possibilities for signal playback. Known workaround solutions are contrary the cost advantage. Voicecoil actuators’ permanent magnets are primarily leading to high cost. However, bipolar actuation greatly simplifies control-related integration, I would rather say even makes it possible. In the case of favorable design of the frequency response and corresponding integration of its transfer function in the control concept, is enabling the realization of a variable high-quality haptic feedback on the surface.

1.4 The 2020’s S-Class Center Information Display and the EQS’MBUX Hyperscreen

The 2020’s S-Class as well as the 2021’s EQS (BR223 and BR297) are representing the upper end of Mercedes-Benz’ automotive quality and innovation. Thus introducing premium displays with OLED technology and 3D instrument cluster displays on a new level, cannot miss haptic feedback to support the driver’s interaction with this system.

At all, generating a haptic feedback on a 12,8 “display or even a 17” (EQS) display with a length of more than 1,40 meter and depending to this, a high mass of several kilograms and a not moving surface requires different approaches compared to the previously described principles.

1.4.1 The Haptic Concept

In particular, shaking the entire surface with an adequate dynamic behavior as discussed before would trigger an enormous momentum, including disturbing noise. Additionally two users should be able to interact simultaneously on the surface, with each of them receiving a separate haptic feedback.

The Time Reversal method (TRM), as described in [2], uses several actuators to create a specific pulse very precisely at exactly one position. Perhaps the approach seems to be “too perfect”, which is why it gets a disadvantage bringing in a delay of several hundredths of a second, which is contrary meeting user interface requirements of a maximum overall system response.

Another possibility based on the generation of bending waves, which uses natural resonances of a surface. NXT developed and patented this method as Distributed Modal Loudspeaker (DML) to realize audio playback through large surfaces. The continuation of NXT under the firm Redux later also generated haptic feedback on surfaces, today found in Google Pixel 5.

1.4.2 Use of Inverted Transfer Functions

The s-class’ approach differs to the previous ones; it uses the inverted transfer functions between each transducer and the finger position. Folding these inverted transfer functions with the defined vibrotactile broadband target signal to apply at the fingertip. Generating a set of actuator signals for each finger position and each actuator. The application of these actuator signals on the real physical hardware is turning them back to the specific target signal. At the finger position, the single signals of each actuator are overlapping and combining to the previously defined super-wave-like vibrotactile broadband target signal with higher intensity at the specific point.

Figure 14.14 describes the control concept of one single signal path. On the left the source, called target signal $F_S (s)$ is the signal wanted to be playing at the fingertip, i.e. it shall be equal to the signal at the target point $F_{TP}(s)=F_S(s)$. The measurement of the signal path determining by frequency sweep, impulse response or step response. At all, it is important to apply the correct mechanical impedance that needs the inclusion of the finger’s impedance. Comparable to the IDM Measurement as described in [Chap. 13], the impedances are influencing the required transfer functions. Figure 14.15 is showing a dynamic measurement system used for determining the transfer functions as well as for evaluating the haptic feedback of the system in lab and end-of-line.

Fig. 14.14

Control concept of one single signal path

Fig. 14.15

Dynamic haptic measurement on the EQS’ MBUX Hyperscreen. Used for evaluation, end-of-line testing and for determining haptic transfer functions

On this basis, the calculated actuator signal A(s) is stimulating the surface in the way, that real existing physical transfer path $G_{ph}(s)= G(s)$ is now turning back the signal into the original target signal $F_{TP}(s)=F_S(s)$. Of course, there are deviations due to quality of the whole measurement and calculation accuracy as well as environmental and production tolerances. So $G_{ph}(s) \sim G(s)$ is the correct writing. At all, executed tests showing up that no further compensation mechanisms are necessary to run the system in a proper way. That is why we can see $F_{TP}(s) \sim F_S(s)$ as sufficiently for the purpose. This transfer function is considering the whole transfer path, from digital signal definition, via the whole electronics including actuator, surface impedance and of course the finger impedance.

Now, folding each target signal $F_S(s)$ with the inverse transfer function $G^{-1}(s)$ towards each of the actuators derives the required set of actuator signals A(s). Combining and playing the sets associated actuator signals is superimposing to a strong and intense haptic signal the specific finger positions. Of course, it requires the execution of the process for each single target signal.

Figure 14.16 gives an overview about the control concept. It describes the general control principle also including the identification of the transfer functions between each actuator and each position that are used to generate the required actuator signals out of each target signal.

Fig. 14.16

Schematic of determining actuator signals. Also showing the acoustic optimization path

The indices show the dependencies on the number of signals (n), number of actuators (a) and number of finger positions (p). So the number of actuator signals for example is the product of n, a and p. It can be easily understood that there might arise a high number of actuator signals to be calculated.

Two different strategies remain implementing that strategy in practice. First approach the ad-hoc calculation of the actuator signals requires a high computing performance to do all complex math in time. The second approach, instead more memory intensive, is the pre-calculation of each of the actuator signals $A(s)_{n,a,p}$. In general, it makes sense to define the size of the finger position areas to limit their number. The lateral size of the finger position areas depends on the system properties. A general simple assumption is a diameter of 3–7 cm.

1.4.3 Haptic-Acoustical Optimization

Experiencing a control elements’ feedback always is a multimodal perception. The overall impression contains visual, acoustical and haptical aspects. Especially the acoustic and haptic feedback have a physical relation that can easily lead to conflicts. The frequency range of the haptic feedback lies at around 0–1.000 Hz and is overlapping the acoustical range lying around 16–20.000 Hz. This means that the haptic design is interfering with the acoustics between 16–1.000 Hz.

Due to its acoustical sensitivity, around 1.000 Hz unfavorably covers the most sensitive range of hearing. At all, the perceptional weighting reduces the relevant haptic range down to 500 Hz; at all, they still lie closely together somehow complementing each other, at all, the remaining haptic range is still perceived acoustically well and optimizing haptic signals leads to acoustical side effects and opposite.

The basic principle of acoustic optimization strategies are comparable with the previously described haptic procedure: measurement of the transfer behavior, comparing with the target signal and filtering the signal. The transfer path of course is a different one than the haptic signal path.

The differences are huge anyway: while haptics is focusing on one specific surface area, the remaining areas do not really matter. The acoustic feedback instead is radiating always via the entire surface. This means that all vibrations across the whole surface are part of the acoustic feedback, while only the contact area of the finger affects the haptic feedback perception, as shown in Fig. 14.17.

Fig. 14.17

The actuator signal needs to generate the haptic feedback as well as the acoustical perception

Each of the actuators is stimulating the overall surface vibration in a different way, that why, so separate acoustical optimizations for each are necessary adding different acoustical influences to fit the target acoustics. At all, there is also a difference between the optimization of noise and acoustical quality.

Typically, nonlinear behavior of the surrounding environment mainly is responsible for noise. Having its specific resonant frequencies amplifying disturbing noise, reduced by dampers, compensational masses i.e. mechanical strategies or influencing the driving signal like Fourier or frequency shaping strategy as described already. The focus lies on the reduction of the intensity of the disturbing frequencies.

Whereas optimizing the acoustic quality is different, more a designing and composing process of the acoustical sound, for giving a clear feedback, not being intrusive but always expressing quality. Ideally, for this the active system is able to generate a linear acoustic behavior more like a high quality loudspeaker.

The challenge of optimizing haptic as well as acoustic quality and noise lies in the same actuator system driving both modalities. It is limiting the possibilities and efficiency of a system, because the optimization of the acoustical behavior may lead to a worse haptical quality in the same way as a strong haptical feeling may lead to a miserable acoustical behavior.

Each actuator can be of different effectiveness in each frequency band due to the individual acoustical and haptical transmission functions. To achieve efficient optimization, the analysis of the contribution of each actuator needs to perform separately for each required frequency band. Thus, for example, an acoustically less effective actuator can generate higher haptic contribution than another one, while in turn the other one may be primarily useful for acoustic optimization. Figure 14.18 shows the scheme for a frequency-specific optimization of all actuator signals to get a uniform acoustical feedback combined with a uniform haptical one.

Fig. 14.18

Acoustic band selective optimization for each actuator and each target signal. The colors of the bars show the fraction of each actuator in each frequency band. The previously only haptically optimized actuator signals are now changing for also optimizing acoustical impact of the feedback signal. Not to misunderstand: it is not a mix of haptic and acoustic signals, but an adjustment of the actuator signals to get an ideal combination of both aspects

The acoustic measurement of the signals shall take place with a coupled finger impedance and ideally take the user’s typical distance to the display into account. Figure 14.19 shows the measurement, without a mechanical finger impedance. Typically, the optimization process is performed in cycles, Fig. 14.20 shows the optimization progress due to measurements of finger positions based on a similarity index, which expresses the similarity of the signals regarding the mean value of the acoustic parameters.

Fig. 14.19

Acoustic measurement on the EQS’ MBUX Hyperscreen. Used for evaluation and for determining the acoustic transfer functions. Not shown is the finger to operate the device with the correct mechanical finger impedance. The microphone (red circle) is placed in a typical distance of the user

Fig. 14.20

Similarity index before (left) and after (right) acoustical optimization, describing the relation between local and average values

1.4.4 Type and Position of Actuators

What kind of actuator to use, does not depend on the electromechanical principles. The use of solenoid, voice coil types or piezo actuators primarily depends on their mechanical properties like frequency range, force/travel intensity and efficiency as well as thermal stability. At all, there are several actuators available of each type fitting into this scheme of vibrotactile frequency range. Hence, finally building space and cost efficiency are becoming the main drivers for this decision.

The physical efficiency of an actuator is determined by the coupling of the required frequencies into the active surface. Besides the mechanical coupling, that typically is gluing, the dynamical behavior of the system is of importance: Each surface has its own specific vibrational modes, as for example shown in Fig. 14.21, where the frequency and shape of the modes are physically determined for each setup via FEM-simulation. To optimize the efficiency of the dynamical behavior, we take a closer look at the basics.

Fig. 14.21

FEM Simulation of four vibrational modes of the Mercedes-Benz EQS’ MBUX Hyperscreen. Red color shows the maximum amplitude, blue the nodal lines

Even using the whole frequency bandwidth of the signal and not only the single modes of the surface, it is a good approach to use the modes’ information for optimizing the positions of the actuators. Figure 14.22 shows simplified three different actuator positions A, B and C as well as the surface specific Eigen modes 0, 1 and 2. Actuators placed in vibrational node points cannot activate this specific vibrational mode, as an actuator in position A cannot activate the modes 1 or 2 and position C cannot activate mode 2. While position A is an ideal driving position for mode 0, because it is located at the position of maximum oscillation. Position C seems to be efficient for mode 1 but only moderately driving mode 0. Looking at position B, it shows a strong excitation of each mode, that why it would be the preferred position. In practice, unfortunately, not every actuator can stimulate every mode, but an optimization as explained makes sense in any case to become most efficiently.

Fig. 14.22

Simplified illustration of vibrational modes 1–3 of a surface and 3 specific positions A-C for actuators

1.5 Controlling the Haptic Feedback

Physically generating a good haptical feeling is the one thing, but to control the feedback appearing at the right time, place and in the right sequence is crucial. Especially when applying complex haptical combinations of different haptical bricks on small space, i.e. search haptical effects like edge effects and texture, shall appear instantly, or several push feedbacks shall instantly within short time.

The interpretation of finger positions and forces are a key feature followed by a performant activation and effect collision handling, prioritizing or mixing different haptical effects. Thus, even at permanently dynamical changing screen layouts, the generation and interpretation of haptical definitions can lead to a high level of complexity. Figure 14.23 describes an interaction process considering basic mechanisms for feedback control. First, the user is receiving information via the visual screen content and he is deciding to do some finger interaction. Receiving this finger interaction, e.g. sensing touch and force values, the systems is interpreting this user’s input. It is recognizing for example some instructions that lead to an approval in form of a haptic feedback and needs to change the state of the system’s user interface, for example changing the screen content with updating the haptical definition and of course maybe activating a specific function. If several feedback effects collide, the system needs to be able to prioritize and maybe mix effects, for example a texture and a click effect.

Fig. 14.23

Interaction process including the activation of haptic feedback

Perceiving the whole user interface as helpful and of high quality by the user, besides the physical quality of the feedback, it requires uniform integration of haptical concepts as well as powerful concepts enriching haptical experiences by combining different effects to a new overall impression. This may shift haptics to a next level.

2 HapCath—Haptic Catheter

• Thorsten Meiss,
• Nataliya Koev,
• Thomas Opitz,
• Tim Rossner &
• Roland Werthschützky 

2.1 Introduction

Catheterization is a medical procedure used for diagnostic and therapeutic treatment of blood vessels. For example arteries of the heart suffer from atherosclerotic depositions which diminish the blood flow and, as a consequence, result in heart pain, heart attack and heart failure. In the US, diagnostic and interventional catheterizations of the heart were performed approx. 1.5 million times in 2014 (Data base 2021, diagnostic and therapeutic counted separately [27]). In many cases, catheterization is a simple process for well-trained cardiologists: a guide wire is inserted into an artery, usually the fermoral artery at the upper leg, and is slid towards the heart. By rotating the proximal end of the wire (the end in the physicians hand), the physician leads the tip at the wire’s distal end into the coronary arteries. To visually control the guide wire movement, short time 2D-X-ray video is used. By sliding a catheter over the guide wire, the physician can lead contrast fluid into the vessels to visualize the course of the arteries for diagnostic purposes. Through this hollow catheter, the physician can lead tools to the upper branches of the coronary vessels or change and reposition the guide wire very quickly. To reopen totally closed or occluded vessels, the physician can thread a balloon catheter over the proximal end and slide it over the guide wire, through the catheter, into the occluded part of the vessel. Then the affected vessel can be widened by inflating a balloon (dilatation) and optionally expand a stent to prevent the vessel from contracting again.

However, in many cases, the vessel is totally closed and penetrating the occlusion with the guide wire tip becomes very difficult. Additionally, navigating the wire through calcified and contorted vessels often turns into a challenging task. The flexibility of the guide wire has to be adapted either to follow contorted vessels or to penetrate occlusions. Therefore, the wire has to be exchanged during the intervention. The risk of penetrating the vessels increases. The intervention becomes time consuming and sometimes even impossible.

A reason for prolonged catherization time is caused by the limited feedback of the guide wire. Due to the small diameter and the necessarily low stiffness of the wire, the physician cannot feel the forces at the guide wire tip; only 2D-X-ray imaging linked with the legal limitation of the amount of noxious contrast fluid, requires well trained operators and a long training phase of new cardiologist. It needs experience to match the limited visual feedback with the real motion of the guide wire. To overcome these challenges originating from a lack of intuitive usable information from the guide wire’s tip, the HapCath system provides haptic feedback of the forces acting on a guide wire’s tip during vascular catheterization [9, 15] (Fig. 14.24). In order to achieve this, force measurement and signal transmission out of the patient’s body is realized. Thus, the transmitted signals are used to control actuators of a haptic display to provide a scaled, amplified force, which is coupled back onto the guide wire. This scaled force surpass the friction force of the guide wire, enabling the user to feel the tip interacting with the walls and obstructions inside the vessel. This shall simplify and accelerate the navigation of the wire and reduce the risk of punctuating the vessel or damaging and striping of ateriosclerotic depositions. The aim of providing haptic feedback is to enable grasping the right way through the vessels just like with a blind man’s cane. For this purpose, very small force sensors have been designed, fabricated, tested and integrated into guide wires. A special electronics to calculate the 3D-force vector acting at the tip has been designed and a haptic display with a translational and a rotational degree of freedom to couple the amplified forces back onto the guide has been constructed and tested.

Fig. 14.24

Schematic of the assistive system HapCath: The forces $F_0$ at the tip of the guide wire are measured by means of a small force sensor. The signal $S_{F0}$ is transmitted out of the patient’s body over the wire. Within a haptic display the signal $S_{F0}$ is reconverted into a scaled force $n \cdot F_0$ by means of amplifiers and actuators, thereby overcoming the friction force $F_F$ within the catheter and vessels. This force is displayed to the surgeon’s hand as the amplified force $F_H$

2.2 Deriving Requirements

To our knowledge, the exact forces at the guide wire tip during catheterization have been unknown up to recently. For this project detailed analysis of the advancing of the guide wire within the vessels with simulation [8] and experimental measurements [12] of the guide wire interactions have been performed.

2.3 Design and Development

2.3.1 Force Sensor Design

The Fig. 14.25 shows selected relevant scenarios of the interaction of the guide wire with the vessel walls and with stents inside the vessel.

Fig. 14.25

Pictures of different interactions of the guide wire with the vessel, with different kind of plaque (a), within a heavily wriggled vessel path (b), and with a stent (c) and (d)

A force sensor can be integrated at the tip of the guide wire or with some distance to the tip. To allow for the measurement of the interaction forces when the guide wire is advanced with the tip backwards (Fig. 14.25d), the sensor needs to be integrated with several centimeters distance from the tip. This will lead to additional friction forces in the sensor signal and will result in lower frequency response due to higher mass and damping. The most beneficial location to integrate a force sensor therefore is directly into the tip of the wire, due to higher amplitude and frequency resolution of the contact force measurement.

Simulations have been performed were the guide wire is modeled as distributed elastic elements interacting with viscous elastic walls of the arteries with Matlab©[8]. Additionally, experiments to determine the buckling load of different types of guide wires where conducted [12]. Both methods reveal a maximum force in axial direction of the wire of around 100–150 mN, e.g. for penetration occlusions. This is depending on the type of guide wire used. The forces during advancing and navigation and detecting surface properties, e.g. roughness or softness, are estimated to be in the range of 1 mN to 25 mN.

To allow for force measurement at the tip, two types of micro force sensors have been designed and fabricated [10, 12, 13, 19] (Fig. 14.26). They are built from mono crystalline silicon with implanted boron p-type resistors. This technology was chosen to fulfill the requirements on micro scale manufacturing with its high level of integration, a relatively high voltage output for robust external readout as well as high mechanical stiffness to fulfill the requirements on high frequency resolution up to 1000 Hz. For stable control of even very low forces and for safety reasons, it is required to support static force measurements. This allows to keep exact force control of low forces even after the guide wire tip was in static contact with a constriction over longer period of time.

Fig. 14.26

Two types of mono crystalline silicon force sensors; both are designed to resolve the full force vector in amplitude and angles. Their size is compared to an ant

2.3.2 Guide Wire and Sensor Packaging

Guide wires are disposable medical products, manufactured with technologies of precision engineering. The guide wire requires a maximum torsional stiffness and a variable bending stiffness along the wire. To integrate an electrical connection of the sensor over or within the guide wire is a challenging task. A loss in rotational stiffness due to softer materials of the conductors than stainless steel or nickel-titanium will result in less mechanical performance. This is the main reason why the space for the integration of electrical wires is very sparse. The electrical connection is established with four insulated robust copper wires, each with a small diameter of 27 $\upmu$m.

The sensors are glued onto the tip of the wire with UV-curable medical adhesive. First prototypes where enclosed into a flexible polyurethane polymer [14] and covered with medically compatible Parylene C. The second generation of tactile guide wire is covered with Pebax 3533, and hydrophilically coated [25]. This improves navigability due to high lubricity and reduced friction of the guide wire. The Fig. 14.27 gives an insight into the assembly of the wire of the second generation.

Fig. 14.27

The integration of the sensor into the guide wire tip encompasses several steps of precision mounting, dispensing and covering with glues and cover polymers

2.3.3 Haptic Display Design

The guide wire is navigated through the vessels with two degrees of freedom: translationally, to advance the guide wire, and rotationally, to choose the relevant wire branch. The haptic display is designed to provide these forces and motions (Fig. 14.28) [22].

Fig. 14.28

Basic design of the haptic user-interface with the translational degree of freedom and side view of the implementation [22]

The haptic display supports the generation of static forces to display the penetration of occlusions. To give feedback of surface roughness and to reflect the dynamic amplitudes during penetration or when the wire is moved over the grid of stents or rough depositions, the haptic interface is designed with low mechanical inertia to generate high frequency feedback as well. To optimize the dynamic performance of the haptic interface the equivalent circuit representation of the electro-mechanical setup with guide wire and passive user impedance is used (Fig. 14.29) [22].

Fig. 14.29

Equivalent network representation of the haptic user-interface including the guide wire and the user’s passive mechanical impedance

Because the friction force is a relevant factor of all forces, it is assumed that further optimization of the haptic impression can be achieved by reducing or canceling out the friction forces. The Fig. 14.24 shows that the force at the physicians hand $F_H$ at interface equals the sum of contact force $F_0$, the friction force $F_F$ and the force $nF_0$ generated by the haptic display. In turn, the friction force $F_F$ can be calculated when the contact force $F_0$, the force $nF_0$ of the haptic interface and the force $F_H$ at the hand of the physician is known. Therefore, the hand force $F_H$ needs to be measured in real-time. To allow this, a hand force sensor is developed and tested. The Fig. 14.30 shows the technical implementation and integration of this sensor into a handle part, used for steering a guide wire during catherization.

Fig. 14.30

Hand sensor integrated into a handle for steering the guide wire. FEM Analysis for a tensile and b compressive force. c integration concept of the hand force sensor to assemble a standard size handle for guide wire manipulation

2.3.4 Electronic Design

The sensor and haptic display is powered with one single electronic system. The electronics provides power supply to the force sensor and a unique six channel high resolution analog frontend. A micro-controller unit is used to control the sensor readout and to calculate the 3D-force vector of the contact forces. It provides angle measurement and PWM-control for two brushless DC motors for the haptic feedback interface. Force signals are transferred to a PC and to a display for information purposes. The control loop is implemented in the micro-controller itself without the need for time critical communication with the PC. This allows for a fast control loop with up to 10 kHzs${^{-1}}$ control rate for smooth haptic feedback [12]. A second design is developed using LabView real-time system [21].

Fig. 14.31

Moving the guide wire with a minimal contact force. The physician maneuvers the guide wire by moving the handle by rotating, pushing and pulling (a). Measurements with a first generation prototype of the tactile guide wire within a model of the arteries with artificial plaque (b). The measurement shows the sensor signal during inserting, moving the wire forward, rotating the tip and going into the right and then into the left vessel branch

2.4 Verification and Validation

To validate the function of the overall haptic system, the tactile guide wire, the electronics and the haptic display are connected [15]. The guide wire is advanced into a model of the arteries build from silicone tubes. The clean silicone tube surfaces mimic smooth, healthy arteries. Partially, the tubes are filled with epoxy glue mixed with sand, which mimic rough depositions like calcified plaque. The Fig. 14.31 shows the sensor signal when the guide wire is moved inside the model. The test signals are fed back to the haptic display and the interaction of the tip with the vessel walls can be discriminated. To analyze the force response in more detail, glass plates with different surface roughness have been prepared and the guide wire tip is moved over the surfaces and the signal is recorded and presented using the haptic display (Fig. 14.32). The resulting haptic feedback allows for discrimination of smooth and rough surfaces. When touching different surfaces the contact force is amplified and different surfaces can be distinguished clearly.

Fig. 14.32

Force signal over time recorded from packaged sensors integrated into a first generation guide wire prototype to evaluate different surface roughness. Glass (a) paper (b) sand in epoxy glue (c). Notable is the reproducible, nearly periodic output of the packaged sensor on paper (b). Increasing roughness of the surfaces lead to increasing output signals (a), (b) to (c)

By increasing the amplification factor of the forces, the sensation of soft or elastic surfaces change to that of much more rigid features due to the higher stiffness emulated by the system. This makes soft, fragile surfaces much easier to detect. They become virtually harder, whereby the elongation of the tissue due to force is reduced. We assume that this can lead to much less ruptures of vulnerable features, leading to fewer complications during catherization in the future, as well as a higher success rate for complicated interventions with wriggled arteries.

2.5 Design Updates and Lessons Learned

The project HapCath, Haptic Catheter, was started in 2004 and several challenging design tasks have been supported with research topics. In a project to transfer research to application a second generation of guide wire prototypes have been designed together in cooperation with industrial partners. Research was conducted to allow sample manufacturing of guide wires and to improve stability of the signal transmission. The sensor design has been adapted. The main requirements derived in the early project phases remain valid.

To extend the experiments and to validate the findings in a realistic test environment all the technical components have been set up to a new demonstrator system (Fig. 14.33). This system is used for demonstration and tests with vascular surgeons in order to demonstrate and to optimize the system performance.

Fig. 14.33

Tests inside a standardized vasular model [10]

Senior cardiologists performed tests on navigability of the prototypes in standardized vascular models (Figs. 14.33 and 14.34). They report good characteristics of the second generation prototypes. To meet full performance characteristics of a high performance recanalization guide wire, the core stiffness can be increased by higher cross-section.

Fig. 14.34

Tests inside a standardized vasular model (a) and (b). The forces measured with an external force sensor are increasing with the insertion depth of the guide wire due to overlaid friction forces. Only with the integrated force sensor and a haptic display the contact force at the guide wire tip can be made haptically available.[10]

2.6 Conclusion and Outlook

The project involves several technical challenges, encompassing sensor design and sensor integration, as well as adapted haptic feedback. The current field of research is focused on transferring the results into application by refining the design and the technical implementation of the sensor, electrical wire and guide wire assembly. The test with cardiologists and optimization of the guide wire is ongoing. The application will benefit from ongoing research regarding optimized filtered signal feedback from touch scenarios of the tip with different surfaces. Additionally, research on smart wires is performed where the orientation of the guide wire tip and the stiffness can be controlled electrically by using smart materials. The HapCath project is funded by the German Research Foundation DFG under grant WE2308/3-1:3 and WE2308/15-1:2.

3 FingHap—Haptic Finger Rehabilitation Device

Alireza Abbasimoshaei, Thorsten.A. Kern, Yash Shah

Institute of Mechatronics in Mechanics, Technische Universität Hamburg

al.abbasimoshaei,t.a.kern@tuhh.de

3.1 Introduction

Lack of rehabilitation services in rural areas has been one of the major issues that make a big difference in facilities through different living areas. The main reasons found are the availability and reachability of the physiotherapists in different areas. Rehabilitation is one of the most important procedures that should be done after injuries. Due to the repetitive nature of this training, a full robotic system could help the physiotherapists to rehabilitate a larger number of patients either through home rehabilitation or wearable vibrotactile systems  [24]. Such a device can record data or can be used as a live system. It could be consists of the operator-device, patient-device, and haptic mechanism. In this section, after a brief introduction of the design and controlling system, a telemanipulation system based on the ROS system is introduced. This section presents the FingHap rehabilitation device that helps patients to stay at home and still acquire the same level of treatment. The patient and physiotherapist will be connected through a cloud server, and a proposed external bi-way communication approach has been applied between them. The therapist would physically guide the robot present at the clinic, and as real-time communication, the patient’s robot would replicate the same motion. As a matter of feedback, the state of the patient’s robot will also be passed on to the therapist’s robot. In this way, a therapist can physically experience the patient’s feelings and decide the next level of exercise. All control parameters like velocity, force, and PID values of the patient’s robot can be accessed and controlled by the therapist. The approach has been tested and the achieved results are shown.

3.2 Design and Prototyping

In this robot, all DOFs of finger joints and wrist are moving by two motors. A schematic view of the designed system with a hand is shown in Fig. 14.35a. As it is shown, the green part that is the finger placement is a flexible system to adapt the rotation according to the joints’ center of motion.

As can be seen in Fig. 14.35a, the hand is located in the upper section that includes the green finger part and two ball bearings. Because during the rotation, the joint’s center of motion changes, a flexible system is used for the finger part. Due to the flexibility of this part, fingers can be rehabilitated by adapting the center of rotation.

This system has two motors used to move the cables and provide the fingers displacement (Motor1) and wrist rotation (Motor2). Moreover, two ball bearings and a shaft are used to transmit the motor rotation to the wrist part [4, 18].

Some tracks are designed to adjust the length of the finger parts according to each patient (Fig. 14.35b). Figure 14.35b shows index finger DIP training configuration. As it can be seen, it contains the finger part and the finger joints movements are restricted by a bar at the back of the fingers. Also, this bar contains a circular part at the end to make the adjustment and unlocking different joints easier. Thus, changing the unlocked joint leads to different phalanx rehabilitation. There is a bar at the backside of the system to lock or unlock the joints. The configuration of the bar shown in Fig. 14.35b is for DIP training of the index finger. Figure 14.35c shows the posture of the index finger in the device. As it is shown, the finger part is adjusted according to the length of the finger, and DIP is placed at the tip of it. The movement of the index finger is provided by cable shown in Fig. 14.35c and the spring moves it back (Fig. 14.36).

Fig. 14.35

Schematic design of the system

Fig. 14.36

Prototype of the rehabilitation robot

3.3 Design an Adaptive Fuzzy Sliding Mode Controller for the System

3.3.1 Desired Trajectory During Rehabilitation

To design a useful control system, it is needed to find the desired trajectory of each part. Thus, for finding the desired trajectory of fingers, the movements kinematic of all of them were analyzed during their tasks. Ten healthy subjects, seven males and three females did finger exercises in three different velocities [16]. Every subject performed five trials for each joint and they rested about one minute between the trials. For each trial, the physician fixed their finger joints and they moved the free joint. They moved their phalanx according to the physician’s instructions and an attached gyro sensor measured the angle of rotation. These patterns are independent of finger length because they depend on the joint angle and time.

The average of the collected data was found and fitted with a polynomial. Before fitting, the average method was used to remove the noise from the data. The equation for the DIP joint’s rotation angle of the index finger in flexion is as (14.4) and it is a second-order polynomial and Fig. 14.37 shows the fitting graph with 0.9788 R-square (with $K=1$). Equation 14.5 and Fig. 14.38 shows the results of fitting a third-order polynomial with $K=1$ to the PIP joint with 0.9857 R-square.

Fig. 14.37

DIP angle with respect to time (with $K=1$)

$$\begin{aligned} \theta = {K} \times (0.27 \times t^{2} + 2.4 \times t + 0.5) \end{aligned}$$

(14.4)

Fig. 14.38

PIP angle with respect to time (with $K=1$)

$$\begin{aligned} \theta = {K} \times (-0.23 \times t^{3}+3 \times t^{2} + 0.67 \times t - 2.2 ) \end{aligned}$$

(14.5)

Such works were done for other fingers and phalanges. The equation for the MCP joint of the index finger is as following.

$$\begin{aligned} \theta ={K} \times (-0.41 \times t^{2} + 14 \times t - 2) \end{aligned}$$

(14.6)

The patients are trained under the supervision of a physician at different speeds. After recording the average data, it is found that the medium and fast speeds of training are reached by multiplying the slow speed ($K=1$) by $K= 2.2$ and $K= 3$, respectively.

3.3.2 Mathematical Model of the System and AFSMC Design

By using Newton’s law on the fingertip the dynamic equation of the system is found [17].

$$\begin{aligned} \begin{aligned} {I}{\ddot{\theta }} = T \times \sin (\alpha )&\times l_{3} + T \times \cos (\alpha ) \times E- K \times ((\sqrt{A}-\sqrt{B})\times \cos (\beta )\times l_{3}\\&+ (\sqrt{A}-\sqrt{B})\times \sin (\beta )\times G)- C {\dot{\theta }} - {K}_{1} {\theta } \end{aligned} \end{aligned}$$

(14.7)

$$\begin{aligned} A=({H}+l_{3} \sin (\theta ))^{2}+ (l_{1}+l_{2}+l_{3} \cos (\theta ))^{2} \end{aligned}$$

(14.8)

$$\begin{aligned} B={H}^{2}+(l_{1}+l_{2}+l_{3})^{2} \end{aligned}$$

(14.9)

$$\begin{aligned} {I}{\ddot{\theta }} = T \times R \quad . \end{aligned}$$

(14.10)

The robot’s simplified kinematic model is shown in Fig. 14.39a and E, G, and H could be seen in Fig. 14.39b. In Eq. 14.7, $l_{1}$, $l_{2}$, and $l_{3}$ are the length of the phalanges, I, R, and $\theta$ are the inertia of the rotating part, the motor shaft, and the rotation angle of the finger respectively. Also, C, ${K}_{1}$, and K illustrate the robot’s damping and stiffness and the spring’s stiffness. I shows the moment of inertia and T is the cable force. $\alpha$ and $\beta$ are as below and D shows the distance between the finger part and the connection point of the cable with the system (Fig. 14.39b).

Fig. 14.39

Simplified kinematic model of the robot

$$\begin{aligned} \alpha = \theta +\arctan (\frac{D-l_{3}\sin (\theta )-E\cos (\theta )}{l_{1}+l_{2}+l_{3} \cos (\theta )-E\sin (\theta )}) \end{aligned}$$

(14.11)

$$\begin{aligned} \beta = \theta + \arctan (\frac{l_{1}+l_{2}+l_{3} \cos (\theta )+G\sin (\theta )}{H+l_{3} \sin (\theta )-G\cos (\theta )}) \end{aligned}$$

(14.12)

In Eq. 14.7, $\theta$, l, x, and y are the finger rotation angle, the cable length, the horizontal, and vertical axes of the cable length respectively.

For removing the unknown parameters and uncertainties effects in the system mechanical model identification, a Sliding Mode Controller (SMC) has been designed. This controller reduces the effects of parameter variations, uncertainties, and disturbances.

The Sliding Mode Controller (u) consists of two controllers (${u}_{eq}$ and ${u}_{rb}$) and it should guarantee the stability of the system. The equivalent controller is ${u}_{eq}$, ${u}_{rb}$ controls the uncertainties and disturbances, and they are considered as Eqs. 14.14 and 14.15. More detailed information on this system’s Sliding Mode Controller is brought in [17].

$$\begin{aligned} {u}={u}_{eq} + {u}_{rb} \end{aligned}$$

(14.13)

$$\begin{aligned} {u}_{eq} = g ^ {-1} ({\ddot{{x}}_d}-f- k (\dot{{x}} - \dot{{x}}_d) - \eta {s}) \end{aligned}$$

(14.14)

$$\begin{aligned} {u}_{rb} = -g ^ {-1} {\rho }. \text {sgn}{({s}) } \end{aligned}$$

(14.15)

Where $\eta$ is a positive constant and the general equation of the system is considered as Eqs. 14.17 and 14.18 in which $\lambda$ is unknown disturbances satisfying Eq. 14.16.

$$\begin{aligned} \Vert \lambda \Vert <\rho , \end{aligned}$$

(14.16)

g and f formula for this system would be obtained as Eqs. 14.19 and 14.20.

$$\begin{aligned} {\ddot{x}} = f({x},t) + g({x},t){u} + {\lambda } \end{aligned}$$

(14.17)

$$\begin{aligned} {y} = {x} \end{aligned}$$

(14.18)

$$\begin{aligned} g =( \frac{1}{I}) (\sin (\alpha ) l_{3}+ \cos (\alpha ) E) \end{aligned}$$

(14.19)

$$\begin{aligned} \begin{aligned} f&= ( \frac{1}{I}) (- K \times ((\sqrt{A}-\sqrt{B})\times \cos (\beta )\times l_{3}\\&\quad + (\sqrt{A}-\sqrt{B})\times \sin (\beta )\times G ) - C {\dot{\theta }} - {K}_{1} {\theta }) \end{aligned} \end{aligned}$$

(14.20)

Where g(x, t) and f(x, t) are unknown functions of the system dynamic equation. Moreover, $\lambda$ is unknown disturbances satisfying Eq. 14.21.

$$\begin{aligned} \Vert \lambda \Vert <\rho \end{aligned}$$

(14.21)

There is a chattering phenomenon in SMC because of the sign function in its formula. In [17] a fuzzy controller is used for solving this problem. Thus, in the previous work [17], a fuzzy sliding mode controller (FSMC) is designed by combining a fuzzy controller with an SMC.

Because of various amounts of stiffness in the patient’s hands, different interaction forces were created between robot and patient. Thus, as another step, an adaptive controller is designed [1]. This adaptive law estimates the uncertainties and the interaction force and drives the error of the trajectory tracking to zero. Thus, (14.22) shows the formula of the designed adaptive controller by considering unknown parameters and patients’ interaction force.

$$\begin{aligned} {u}_{ad}=-\frac{\int {\frac{s}{I}}}{I \times {g} } \end{aligned}$$

(14.22)

In which S is the sliding surface.

3.4 Cloud Enabled Communication

In this project, a trusted google cloud server has been utilized.In this system, neither of the devices has a server installed in it, so they are just clients of the cloud server. It helps decreasing system load to a much larger extent (Fig. 14.40).

Fig. 14.40

ROS—Cloud architecture, motor-images © Dynamixel, used with permission

Moreover, a node in Raspberry Pi acts as a cloud publisher here. Inside the node, a ROS subscriber is formed to extract the data from ROS topics. It is publishing the same data to the server with a publishing rate of 35 Hz or less. The reason is that both ROS and Dynamixel workbench publish the data at 150 Hz and that is too large for a cloud application. Only the important data of the change in the state is transferred from one robot to another.

3.5 System Setup and Data Communication

3.5.1 Passive Mode

Passive mode is considered as the beginning of the rehabilitation therapy and the patient’s hand is supposed to be too weak to do the exercise. Through the robot at the clinic, the therapist would teach exercises to the patient. At this particular stage, only required force is supposed to be inserted on the patient’s hand in conventional therapy. Therefore, from the beginning, a low static current is induced on the motor. In this manner, a static torque is produced by pushing the patient’s hand to do the exercise. To replicate the motion, the current position of the therapist’s robot would constantly be transferred to the patient’s robot. One-way communication is making sure one robot follows another. On the other side, the current position of the patient’s robot is also constantly feeding back. Every time it would be subtracted with the current position of the therapist’s robot and the parameter of difference in position would be found.

Fig. 14.41

Passive therapy structure

If the difference in position would be larger, that would increase resistive torque as per equation in Fig. 14.41 on the motion of the therapist. This resistive torque guides the therapist to reach the position of the patient and it settles down once they both are in the same position and the difference is zero. For further process, the torque which was kept static on the patient’s side can be accessed and increased. This torque parameter is iterable and can be changed to make the patient follow the motion.

3.5.2 Assistive Mode

Assistive mode is considered as the second stage of the therapy where the patient can start the exercise but needs some assistance in between. As the patient begins the motion, the therapist’s robot follows it and as soon as the patient stops, the therapist can provide physical assistance by moving the robot. As per the motion of the therapist, assistive torque (14.23) would keep on increasing on the patient’s hand and it makes the forward motion of the patient possible. This torque has been given an upper limit to avoid putting excessive force on the patient. Similarly, like the first step, if the patient is not able to follow, the resistive torque leads the therapist to reach the position of the patient. In (14.23) $\alpha$ is a constant.

$$\begin{aligned} assistive\_torque = \alpha * torque\_inserted_\_by\_therapist \end{aligned}$$

(14.23)

3.5.3 Active Mode

In the active mode, the patients can perform the exercise by themselves and the therapist can visualize the performance through the robot at the clinic. The stiffness of the patient’s movement can be manipulated through the stiffness parameter to increase the level of difficulty.

3.5.4 Resistive Mode

In the last stage of the therapy, the patient is supposed to already have gained over 80% of recovery. Now, real stress on the arm is required to reach 100% recovery. So now, the therapist provides high resistance to the patient’s movement. In between the free-flow of the patient’s motion, the therapist would hold the robot. The patient’s robot would be rigidly forced to match the same position as of therapist and provides severe resistance to the patent’s onward motion. This resistance (14.24) keeps on increasing as the patient moves forward. The patient has to overcome increasing resistive_torque to keep moving forward. In (14.24) $\beta$ is a constant.

$$\begin{aligned} resistive\_torque = \beta * difference\_in\_position \end{aligned}$$

(14.24)

3.5.5 Automatic Mode

An automatic mode is an alternative to the passive mode. The therapist can directly provide an initial velocity to the patient’s robot with a low static current through the cloud server. The therapist’s robot would follow the movement. On the monitoring, this velocity can be manipulated. If a lot of jerks are experienced in the motion, PID values can be changed on the patient’s side through the cloud server to reduce the jerks. Each parameter can be manipulated in the communication and the desired state can be achieved. Changing the mode is also possible online.

3.6 Experiments and Results

3.6.1 Control System

In an experiment for exploring the AFSMC performance, the slow movement of each phalanx was tested. Ten volunteers performed the finger exercises with different controllers on the robot and the data average was calculated and fitted with a polynomial. As it is shown in [1], adaptive fuzzy sliding mode controller provides better performance because it reduces the effects of differences between patients. Also, it reduces the chattering effects because of its fuzzy controller. According to the experiments data, using an adaptive fuzzy sliding mode controller (AFSMC) reduces the trajectory tracking average error by 80% [1].

Fig. 14.42

Time lag at different internet speeds

3.6.2 Tele-Communication

The time lag depends on many factors. A small analysis was performed to observe the time lag in the communication system at different internet speeds. The data packets of integer values from 100-113 were published in a “for loop” from one Raspberry Pi to another. Fig. 14.42 shows the results for this particular test and the time at which each integer value was sent and received was noted. These data sets were performed at different internet connection speeds of 27,58, and 79 Mbps, and the average time lag were 85,65 and 45 milliseconds, respectively. The analysis shows that internet speed is a big factor in the communication system.

All the therapies were tested on six different people (four men and two women) and the results explicitly convey the proposed system. Within passive therapy, a low static current was induced on the patient’s robot. As soon as the therapist starts the motion, the patient would follow the same motion with a constant force on the patient’s hand. This force could be provided by changing the current in the motor. According to the xm540-w270 Dynamixel motor manual, when this motor uses 0.3 A and 4 A current, it can provide about 0.4 N.m and 8.8 N.m torque, respectively. For the first case, it is assumed that patient had no problems following the therapist. Results in Fig. 14.43 perfectly relate to the assumption as patients smoothly follow the therapist and both of the trajectories in the position versus time graph aligns with each other and they just separated with a small time lag. The Current vs Time graph shows that, though the therapist puts much force on the rehab device, the patient would still experience the low static current that was applied at the start of the therapy. The time lag measured in the experiment was 183 ms and this time lag includes cloud transfer time, besides, to set the received values from the cloud to the patient’s motor and the motor to reach that particular therapist’s position. So this was the overall point-to-point measured time lag between therapist and patient (Fig. 14.44).

As in the more general case, the patient would find certain difficulty following the motion. That leads to an increase in the difference in position between therapist and patient shown. As the difference increases, resistive torque also piles up to the therapist robot as Fig. 14.41.

Fig. 14.43

Typical recording for a passive therapy run

Fig. 14.44

Passive therapy with lag of the patient

Furthermore, the results of the assistive therapy are shown in Fig. 14.45. The patient would start the exercise and the therapist would follow the same motion. As seen in the position vs time graph, at the time counter of 4130 the motion of the patient seems to have stopped. It was reported by the therapist and assistance was provided from there onwards. The assistive torque on the patient’s hand is depicted as the current of the motor in the current vs time counter graph of Fig. 14.45 and it is calculated as per (14.23). High peaks in the current show the assistive torque provided to the patient.

Fig. 14.45

Typical recording for an assistive therapy run

Fig. 14.46

Typical recording for an active therapy run

The active stage of the therapy is shown in Fig. 14.46. The patient is ought to perform a free-flow motion as shown in the position graph. The therapist would increase the stiffness of the motor measured in the graph through the present current of the motors. An increase in the current shows the increase of the motor stiffness and the patient would have to put more power for rotating the robot. The measured time lag from point to point was 145 ms. Results of resistive therapy are shown in Fig. 14.47. The patient has to overcome severe resistance defined as (14.24). Also, in this graph, the resistive torque was defined with the present current of the motor. In all other previous therapies, the average current measured was around 60-70 mA, but in the resistive therapy, the patient has to overcome the resistance as high as 200 mA for moving forward. This is considered as last and the toughest stage of the therapy.

However, most of the position error is because of time lag but the observed error is not relevant to the training success, as the therapist can compensate for a potential lacking-behind of the patient and the success is identified by the amount of travel more than the actual synchronicity of the movement. Thus, there shouldn’t be much difference if the time lag increases by a certain value. Because in this system, all the data is gradually transferred from one part to another. The system makes sure the data sequence remains the same even if there is a high time lag. So the patient would still follow the same exercise but with a high time delay. This is supported by the subjective answers of the patients, that they did neither feel uncomfortable nor disturbed in passive training mode (Fig. 14.48).

Fig. 14.47

Resistive therapy

Fig. 14.48

Intentional time lag

3.6.3 User Experience for Telemanipulation System

The user’s opinions are very important because it should be a user-friendly system. For this aim and to make a comprehensive evaluation of the system, a questionnaire survey was designed for six subjects. They accomplished the survey after the training. All participants had no confusion by using the device and they understood the instructions very well. In the question about the passive mode, it was asked whether the low static current provided by the robot was appropriate or not? 33% said that it was more than enough and 50% asked to increase it a bit. (It is worth mentioning that the static current could be easily changed through the server if required by the patient). According to some reviews, it was needed to reduce the vibration which can be done by changing the model gain of the robot. About assistive mode, active mode, resistive mode, all of the participants said the system performed perfectly as per the defined application and they rated the performance eight out of ten by average, and about half of them insisted to make the motion a bit harder in the active mode. 95% rating was obtained in terms of the safety of the system. About the overall movement of the system, around 84% said that the system is smooth enough and all participants stated that the system works well with a dynamic time lag as well.

3.7 Conclusion and Outlook

In this work, a mechanism for wrist and each joint of fingers rehabilitation with a low number of motors is presented. Moreover, for reducing the unknown parameters and uncertainties effects, an AFSMC design method is proposed. This controller is more robust and independent from the system model because its fuzzy controller output is based on the error. Furthermore, this controller can solve the problems in the controller algorithm from various patient’s differences due to its adaptive part. It was shown that an 80% improvement is observed in the performance of the controlling system.

In another part of this project, a remote supervising system was established. In this structure, a bi-way communication system with a real-time data transfer was developed making one robot follow the other. Thus, the first achievement was itself an effectively working remote supervising system containing a real-time control with active feedback physically realized by the therapist. This approach can be used for any type of rehabilitation device. Additionally, a point-to-point time lag was averagely measured as 145 ms, which is comparatively considered low in the telerehabilitation system. It was possible due to an external combination of ROS and cloud services.

Moreover, the users feeling about the system in different modes were asked by some questionnaires. The overall view of the opinions was good and they pointed some useful statements that will be considered in the next versions. According to the graphs and questionnaire, it is shown that the time lag which is because of sending and receiving the position data, executing the data on the motor, and receiving the feedback, has not a big effect on the performance and a simple camera that gives the physiotherapist an insight into the patient can compensate it. Moreover, most of the participants said that without having this camera, they can do the training without any problem.

This approach can fulfill the requirements for remote rehabilitation in rural areas. The health care system of more than half of the countries in the world is yet to shift from regular therapies to robot-assisted therapies. The cost of developing and delivering robot therapies is very high, but not as costly as the life of people with disabilities. Between all these dilemmas, a system like this can attract and motivate the healthcare industry to look forward to new technologies and modify rehab services.