Evaluation of Haptic Systems

Abstract

In this chapter, a number of measurement methods and tests are presented that can be used either for verification or validation or—sometimes—both. We therefore refrain from an ordering based on this steps, but will present the methods based on the focus of the evaluation method. In that sense one can identify three main foci of evaluation methods: system-centered methods that will test system properties (and are mostly used for verification, Sect. 13.1), task-centered methods that will test the task-performance of a user working with the haptic system (such tests are mainly used for validation but they can also verify system properties depending on the test design, Sect. 13.2) and user-centered methods that will measure the impact of the haptic system on the user. The latter are almost exclusively used for system validation and further described in Sect. 13.3.

Christian Hatzfeld deceased before the publication of this book.

As stated in Chap. 4, the evaluation of a haptic system will test if a haptic system fulfills all requirements defined in the development process (verification) and whether it is conform with the intended usage of the haptic system (validation). In the following, we will focus on the evaluation method itself, the selection of a proper method and the test design is left to the reader. For the design of task-specific haptic interfaces we consider this to be the most practicable approach, because of the uniqueness of the designs. This chapter therefore will not deal with optimized evaluation processes as for example established by Samur for the class of haptic interfaces [46]. This work shows that a more standardized testing of haptic system will exhibit advantages in terms of testing time and comparison of different systems.

Figure 13.1 gives  an overview of the application of task-centered and user-centered evaluation methods with regard to the intended application as described in Sect. 1.5 for the selection of a proper method described in the following sections.

Fig. 13.1

Assignment of evaluation methods to different applications of haptic systems. System-centered evaluation methods are not included for clarity, since they can be applied to almost every system

1 System-Centered Evaluation Methods

The prevalent goal of system-centered evaluation is the generation of comparable technical ratings and values. These are used to verify the developed system against the requirements defined in the development process and to compare different systems with each other. The latter is especially relevant for haptic displays and haptic interfaces, since these systems are intended to be used universally in a variety of applications. Because of that, system-centered evaluation methods are not dependent on a certain type of task or a user. From this it follows that normally there are no different experiment conditions to be considered in the interpretation of such acquired evaluation values.

When designing a haptic system one defines properties of the haptic system based on requirements derived from the application. The assumed properties are usually calculated precisely and are well known in theory. But to confirm the promised characteristics at the final haptic system it is necessary to perform measurements, at least of the most crucial parts. With respect to standard measurement hardware, in the following parts some hints for performing performance measurements are given.

The main focus of this evaluation part is the characterization of force reflecting haptic system. In the different sections values of interest are figured out and some hints are given to implement and conduct the measurements. So, the measurement of a haptic systems workspace, output force- and motion depending values are discussed as well as the measurement of dynamic behavior of mechanical parts and the displayable impedance of different systems. Secondary, special properties of admittance systems and teleoperation systems are discussed.

1.1 Workspace

The workspace, respectively the number and nature of the $\hookrightarrow$ DoF, is the most prominent characteristic of a haptic system. To analyze the work space the measurement of distances and angles can simply be done with a ruler or a measuring tape and an angle meter. Thereby one can differ between active and passive degrees of freedom with corresponding workspaces. Active degrees of freedom are actuated and crucial for the haptic feedback. Passive degrees of freedom show the workspace that is reachable and driven by the user.

Further the characterization can consider the ability to reach every point of the workspace, the independence of the end effectors orientation from the position in the workspace, constraints of the workspace because of singularities, the conditioning number $\kappa$ at each point in the workspace and the global conditioning index $\nu$ (Sect. 8.4).

1.2 Output Force Depending Values

Since haptic systems are bidirectional in energy flux, the verification of output force depending values describes the energy flux directed to the user. Hence the user is seen as passive in this case. The verification of the force depending values can be done in two steps. The first is to investigate static or quasi-static output force signals; the second is to investigate the dynamic output force behavior like frequency response, step response and impulse response.

At first the verification of static force values is shown. Therefore a force sensor, attached to the end effector of the device, with a resolution higher than the minimal desired force output resolution of the haptic system, is needed. The measurement is done by supplying an input to the haptic system and simultaneous inspection of measured output forces.

1.2.1 Static Analysis

 

Peak force:

 is the highest displayable force with very low duration, limited in time due to heat dissipation. When testing the maximum peak force it might be wise to measure the time till the actuator brakes down.

Maximum continuous force:

is the highest continuous force that can be displayed without concerns of thermal destroying of the actuator and is sometimes called as saturation level or saturation force.

Minimal force (offset):

is the lowest force that can be displayed. It is caused by force bias, slip-stick effects, backdrive friction or stiction.

Hysteresis:

 describes the difference of displayed forces during increasing and decreasing device input signals.

Sensitivity:

 is the change of output signal for a change of the device input signal in the linear region. It can also be described as the slope of the graph.

Output force resolution:

is the minimal displayable force step resulting due to minimal change of the device input signal. Thereby the minimal D/A resolution should be considered as well.

Crosstalk:

 describes the influence of an output forces in one direction, produced while displaying a force in a different direction of a multidimensional haptic system.

Dynamic range:

 is defined as the difference of maximum continuous force and minimum force in dB.

  While inspecting the measurement signals one may find different characteristic graphs showing the output force signals in respect to device input signals displayed in Fig. 13.2. The previously described characteristic non-linearities can be derived out of these graphs, called calibration curves [46].

Fig. 13.2

Common non-linearities in the characterization of haptic devices: a saturation, b dead zone, c hysteresis, figure based on [45, 51]

1.2.2 Dynamic Analysis

The verification of static force signals addresses only the actuator and gear capabilities of the haptic system. To maintain the overall mechanical properties of the haptic system, including inertia and damping, it is necessary to characterize the dynamical behavior of the system. One has to note that the movement of the system will induce a unwanted output signal of dynamic force sensors because of acceleration of the mass of the sensor. This systematic error can be dealt with an additional calibration or all measurements have to be conducted in a mechanical fixed condition.  

Frequency response:

 shows the force transfer function between the device input and the measured force response value with respect to frequency. The transfer function is represented by a bode plot and includes amplitude and phase values. With the bode plot one can get information about the useful frequency range of the system, represented as the flat region of the plot without a significant drop, as well as the half-power bandwidth represented by the −3 dB gain drop of the transfer function where the stop band of the system begins. The frequency response is most relevant in control design. Due to the comparison of the waveform of an sinusoidal excitation and the displayed force a measure called force fidelity can be performed to measure the distortion of a haptic signal [21, 46].

Step response:

 represents a time domain analysis method to get quantified information about rise time, percentage overshoot or settling time. The rise time can be derived by measuring the time between the excitation step and the pass of the desired steady state value of the response and is a rate for the maximal displayable bandwidth. Percentage overshoot and settling time are rates for damping in the system and stability. Also information about the force output accuracy in respect to the device input excitation can be derived, represented by the closeness of the output force value to the desired value, as well as information about force precision represented by the repeatability of the output force value and commonly indicated as ± standard deviation.

Impulse response:

 is also a time domain based verification method. With the excitation of a short impulse with a magnitude of for example peak force, it is possible to derive maximum speed and maximum acceleration of the system.

1.2.3 Measuring Conditions

In the most cases it is not possible to provide a universally valid test scenario. Hence before execution of the measurements one should create defined measurement conditions. Therefore it might be crucial to use special constrains to get best fit measurement values for the specific case. The four major conditions to be mentioned are constrains of the end effectors motion and can be figured out as fixed end, open end, hand held and user phantom.  

Fixed end:

means to constrain the user interface to zero velocity. This case might be used for static tests and frequency response without having the influence mechanical properties in the signal. The output force is measured between the haptic system and a rigid clamp.

Open ended:

means a free movement of the haptic system without any constrains. This means zero force at the end effector of the haptic system. In this case velocity or acceleration at the end effector should get taken into account. Therefore an accelerometer could be attached instead or in addition to the force sensor. To derive velocity the acceleration could be integrated.

Hand held:

stands for the users hand and arm gets coupled to the haptic system. The goal is to create mechanical confinements of the haptic system in a realistic manner. Hence it is possible to get measurement result involving all realistic occurring parameters. One should be aware that experiments with human users in the measurement loop vary from trial to trial. For low frequency ranges, humans can adjust their mechanical impedance quite freely, so that such measurements are probably most suited to evaluate the interference liability of the haptic system.

User phantom:

should overcome the disadvantage of undefined conditions by using a human in the measurement loop. With a user phantom, designed to provide almost equal characteristics than a human, it is possible to create acceptable realistic conditions and repeatable results. As a user phantom one can use a rubber with known mechanical properties or in the easiest case a spring and damper combination [21].

  To provide meaningful results and a complete set of measurement values it might be necessary to repeat all measurements for different points and orientations of the end effector in the workspace of a multidimensional system.

1.3 Output Motion Depending Values

Most common for measuring motion values of a haptic system, also multidimensional, is the use of accelerometers. Besides the verification of maximum accelerations, due to integration of the acceleration signals, velocities and due to double integration the position of a specimens end effector can be determined almost without effecting the measurement. For measurements with necessity of high accuracy it might be meaningful to use special sensors for measuring velocity or position. In this case the measurement should be reduced to at least one degree of freedom for the same time.

Measuring motion capabilities can be combined with the measurement of frequency-, step-, or impulse response measurements. Due to the characterization of phase differences of force and motion values one can derive information about inertia and damping of the system [46].

While force depending values are most important for systems based on the impedance structure, for admittance based haptic systems the motion depending values are in the main focus. For a large number of haptic admittance systems the accuracy of displayable velocities or positions is relevant. For example, during the characterization of braille displays the rise time and stationary accuracy of the different pins are of note. To measure the position of small structures for example optical measurement systems like laser triangulators for small unidirectional deflections, tracking systems for large multidimensional movements or vibrometers for highly dynamic movements can be used.

1.4 Mechanical Properties

The achievable haptic quality is direct depending to the mechanical properties of a haptic system. Mechanical properties can vary depending on the position and orientation in the workspace; hence these factors should be evaluated as well.  

Stiffness:

of the mechanical structure is an indicator for maximal displayable stiffness. Thereby the displayable bandwidth is directly limited by the stiffness of the mechanical system. When measuring the stiffness the displacement of the end effector under external force excitation is of interest, while the actuator or the actuators are fixed. The measurement value can be calculated due to the division of the acting force at the end effector and the measured displacement. If the position sensors for the end effectors displacement are integrated to the actuators, there is the possibility to measure the stiffness of the system the other way round. While fixing the end effector with a force sensor one can excite the mechanical structure with forces produced by the actuators of the system. The stiffness can be calculated due to the division of the forces, measured at the end effector, and the virtually measured displacement by the sensors in the actuators. This calculation is equivalent to the slop of the graph till the force excitation limit is reached by the maximum continuous force value of the system.

Backlash:

can be measured by comparing the externally measured deflection of an end effector and the signals of the system integrated sensory. The difference of the signals is a measure for backlash and is usually a problem arising due to the reversal of force directions.

 

1.5 Impedance Measurements

One important measure of haptic displays is the ability to provide a wide range of mechanical impedance. The perceptible haptic impression of a haptic system is given due to a combination of force reflection and velocity. In this case the mechanical impedance means the force reflection of the haptic system in respect to the users inserted velocity. The impedance is a frequency dependent value and can be terminated by $Z=F/v$. Besides the mechanical impedance for one operating point the so called Z-width of the system is of interest [11]. The z-width means the value of impedances that can be displayed by the haptic device. Mostly the development objective of a haptic system is to design the lowest impedance to be like transparent and the largest impedance to be as high as possible to display stiff walls. Since, a haptic systems tends to get instable while displaying very stiff signals, the upper limit of displayable impedances is given by the maintainable stability of the system.

Usually the mechanical impedance gets displayed in a bode plot showing amplitude and phase of a signal. The so generated signals can be used for system identification to create models of the system, showing damping, stiffness and inertia of the system.

For measuring the mechanical impedance of a haptic system one may use a special setup with external force excitation to input a defined periodical force sweep to the haptic system. To calculate the impedance a force sensor and a velocity sensor needs to be attached between the external force source and the specimen. In this case the impedance can directly be calculated as the quotient of the measured force and velocity.

An exemplary measurement setup for measuring mechanical impedance is shown in Fig. 13.3. The main parts are a force source consisting of a DC-actuator coupled to linearly guided rod, an attached force sensor as coupling element between the actuated rod and the specimen and a velocity measurement system. To measure the velocity of the system a laser triangulator with additional analog differentiator is attached.

Fig. 13.3

Measurement setup for the evaluation of mechanical impedances [30]

The shown setup is attached to a hand held haptic controller, based on a delta robot. In this case the impedance measurement is done for the passive system in different directions at the toll center point of the delta robot to examine its mechanical properties. The resulting impedance amplitude is shown in Fig. 13.4 [30].

Fig. 13.4

Measurement of the mechanical impedance of the passive delta structure of the haptic interface of the INKOMAN system from Fig. 13.3

To determine the Z-width one has to measure the impedance of the haptic system when moving in free space to get the lowest displayable impedance of the system. To measure the maximum displayable impedance of the system one can repeat the measurement while rendering a stiff wall with the haptic display by fixing the tool center point with the haptic systems actuators at its maximum continuous force.

If there is no special measurement setup available to excite the system it is possible to jiggle the systems end effector with the humans hand while measuring the force and velocity at the end effector. In this case one is restricted to the dynamic range of the frequency output capability of a human’s hand. Also the calculation of impedances out of excitation values and responding signals for force or velocity independently is possible [46].

Due to the impedance measurement of a system at different points and directions in the working space of the haptic system on can measure the homogeneity or dexterity of the system. Therefore the quotient of the generally smallest and largest measured impedance of the passive haptic system can be calculated.

1.6 Special Properties

Besides the characterization of output capabilities of haptic interfaces one can find different values in complex haptic systems that might be of interest regarding the haptic quality of haptic system.

One of these is for example the input signal of a haptic system that can affect the haptic quality. Hence the bandwidth and accuracy of the provided signals should be larger and more precisely then the set requirements for the displayable haptic feedback. A other point might be the slaves sensing capability of a haptic teleoperation system. The dynamics of the signal can affect the displayable haptic feedback as well as the accuracy of the sensor signals. Also the latency of (force) signals might be of interest and can heavily affect the transparency of the haptic system [22]. Even the time shift between haptic feedback, visual and acoustic feedback might be of interest.

Further properties that have to evaluated depending on system structure and application are transparency and the transparency error (see Sect. 7.5.2), latencies in the control loop (especially when the system contains packet-based information transfer like for example the internet), control stability and energy consumption.

1.7 Measurement of Psychophysical Parameters

Psychophysical parameters like absolute and differential thresholds are fairly well investigated (see Sect. 2.1). Based on the suggestions of Weisenberger et al. it can be nevertheless useful to measure a psychophysical parameter, since deviations to (well-known) thresholds can be attributed to the fidelity of the device [58]. This procedure is implemented in different evaluation testbeds by Samur [46], but can also be applied individually. To assess the fidelity of a haptic system, absolute and differential thresholds are useful measures, evaluating resolution and reproducibility of a device. This method is therefore preferable for the evaluation of haptic interfaces. Similarly, the discrimination of haptic properties and signals can be used to assess haptic systems for communication means as described in the next section.

2 Task-Centered Evaluation Methods

The above described methods of system-centered evaluation are used to determine concrete values for the system’s properties. However, to validate if a system performs correctly, further evaluations of the task performed with the system are needed. Such task-centered methods investigate the feasibility of a system to perform beneficial in the intended usage.

Typical for this purpose is a simple test task that is performed by several test subjects under different boundary conditions. A typical boundary condition is for example the kind of feedback like no feedback, haptic feedback, visual feedback, haptic and visual feedback under which the test is performed or different properties of the haptic system used (for example amplitudes, frequency, speed etc.). Because of the dependence on test persons and different boundary conditions, one should carefully design the experiment to prevent errors because of inter- and intra-personal effects as well as learning.

Results of task-centered method can be used to compare different systems in the same task or to assess the effectiveness of haptic systems in a given application. Furthermore, such tests can be useful to identify promising parameters to optimize a given design or interaction. Starting with these methods, an engineer will take large steps towards the sometimes odd-looking (from an engineers point of view) approaches of human factors and ergonomics. However, the authors find it advisable to consider such approaches as early as possible in the design process (for example by defining evaluation tests in the requirement derivation process) to incorporate noteworthy aspects that will actually contribute to a good product.

2.1 Task Performance Tests

Typical task performance tests are conducted with haptic interfaces using $\hookrightarrow$ VR test setups or with teleoperation system. One should identify a task that is very close to the intended usage of the system to use all interaction primitives involved, but that is simple enough to be understood quickly by the test persons and to be completed in time.

Typical embodiments of such tests are given in the following list.  

Pick and Place Tasks:

Subjects have to pick up an item or object, move it to another location and place the object in a predefined way. Depending on the application, objects and placing rules differ, simple setups only require to place a round peg into a hole, while more complicated setups require to place micro- or nanoscale objects in predefined orientations. Because of the simple implementation, this type of test is often used to evaluate universal haptic interfaces and—of course—systems designed for microassembly [3]. Typical parameters in these tests are the kind of feedback (kinaesthetic/tactile feedback, combinations with visual and auditory feedback etc.).

Tracing:

In these kind of tests, subjects have to follow a given contour in 2D or 3D space or move in given constraints. These kind of tests are especially useful to evaluate the effect of haptic guidance techniques and new rendering techniques in multimodal applications as for example shown in [39], where three-dimensional perceptual deadband coding (see Sect. 2.4.4) was evaluated.

Application Task Primitives:

The third major group of task performance tests are derived from typical tasks in applications. For surgery, this could be suturing or the handing over of a needle or an electrocautery electrode, typical assembly task could include the fastening of a nut or the selection of an object with a certain property. Such tasks can be derived from generalized assessment tests as for example reported in [12] for minimal invasive surgery.

To gain quantified measures from these tests, several outcomes are commonly used. They can be used in combination with almost every test type and have to be chosen according to the goal of the evaluation and the technical capabilities of the test environment.  

Completion Time:

The most basic measure is the time needed to complete a given task. Variations of this test outcome include the number of fulfilled tasks in a given time, sometimes also known as the completion rate.

Errors:

Another outcome is the number of errors made by the test subject. The type of error depends on the task and the intended outcome of the test, commonly used as errors are collisions with other tools or boundaries, the dropping of objects, deviation of the intended placing locations or errors in task performance like the ripping of tissue. Of course, the failing of the task itself can be an error, too.

Handling Forces:

Since one of the main goals for assistive and teleoperation systems is the reduction of handling forces, the evaluation of average, maximum or contact forces is a common outcome of task performance tests. One has to note, that this outcome as well as some of the above mentioned error definitions will require additional sensory equipment like tracking systems or reliable force sensors.

Examples for practical realizations of such tests can be found in a vast number of studies like for example [38], where haptic feedback for robotic surgery is evaluated. Figure 13.5 shows an example using a DaVinci surgical robot. The work of Pongrac giving some general guidelines for the evaluation of virtual reality and teleoperation systems [42]. In the studies included in the meta-analysis about the effects of haptic feedback by Nitsch [40] further task performance tests can be found for various applications.

Fig. 13.5

Pictures courtesy of Peter P. Pott, Technische Universität Darmstadt and Katherine J. Kuchenbecker, University of Pennsylvania

Task performance test setups for minimal invasive surgery: a pick and place setup of a DaVinci surgical robot at the University Medical Centre Mannheim, Germany, b needle transfer setup as reported in [38].

2.2 Identification of Haptic Properties and Signals

One of the main goals of haptic systems intended for communication is to transfer information from the system to the user. Similar is the necessity of a teleoperation to convey enough information to the user to differentiate between relevant components and materials, for example between tissue and vessels in a surgical application. For such kind of evaluations, Tan et al. proposed the evaluation of the $\hookrightarrow$  Information Transfer (IT) of a haptic application, a measure that is used widely in the evaluation of haptic systems [46]. Jones and Tan give a detailed explanation in [28], on which this section is predominantly based.

This approach is orientated on a information theoretical framework and is normally based on a absolute identification experiment. A user is presented one of K stimuli $S_i$ and has to choose a response from a set of K responses $R_j$ with $i,j = 1, \dots , K$. Based on the answers, an confusion matrix is constructed, that denotes how often each response $R_j$ was chosen, when a certain stimulus $S_i$ was presented. Stimuli are represented in rows while responses are given in columns. Based on this matrix, an estimate for the Information Transfer $IT_\text {est}$ can be calculated according to Eq. (13.1).

$$\begin{aligned} IT_\text {est}&= \sum _{j=1}^K{\sum _{i=1}^K{\frac{n_{ij}}{n} \log _2\left( \frac{n_{ij}\cdot n}{n_i \cdot n_j}\right) }} \text {bit} \\ \text {with }&n \text { total number of trails} \nonumber \\&n_{i,j} \text { number of occurrences of }(S_i,R_j) \nonumber \\&n_i \text { row sum} \nonumber \\&n_j \text { column sum} \nonumber \end{aligned}$$

(13.1)

Based on the estimation $IT_\text {est}$, the number of correctly identifiable stimulus levels $n_C$ can be calculated according to Eq. (13.2).

$$\begin{aligned} n_C = 2^{IT_\text {est}} \end{aligned}$$

(13.2)

The upper limit of $n_C$ for a given information channel is also called channel capacity. For haptics, typical values of $n_C$ are in the range of 3 ...4 for unidimensional information transfer [9, 14]. Specialized systems like the Tactuator achieve higher values up to 12 bit [52]. For auditory and visual examples, values of $n_C$ tend to be higher in the range of 5 ...7 identifiable levels of for example force or compliance. To correctly measure the channel capacity, an sufficient high number of stimulus alternatives have to be incorporated in the study. Jones and Tan give a rule-of-thumb to evaluate $K = 2^{IT'_\text {est} + 1...2}$ stimulus alternatives in $n > 5 K^2$ trials to minimize statistical bias and to exceed the maximum channel capacity.

The specification of the information transfer in terms of IT is preferable to the description in terms of percentage correct, that can be found in many studies. The measure of IT is sensitive to chance performance as well as confusions of the response alternatives by the test subject. One has to note, that the information transfer capacity of a channel is considerably lower than the number of $\hookrightarrow$ Just Noticeable Differences (JND) in the same range lets assume. According to Durlach et al., this is due to the fact that information transfer also involves the subjects memory and not only the sensory system [14]. In terms of the general perception primitives described in Sect. 1.4.1, the $\hookrightarrow$  JND measures the discrimination ability of a subject, whereas IT describes the ability to identify a certain stimulus. The latter is much more difficult.

Test setups evaluating the identification of haptic properties and signals are based on real and virtual samples with different object properties (see for example the vast number of experiments with real objects by the group of Kappers [29]) for the evaluation of teleoperation systems and haptic interfaces. A typical experiment could assess the number of different compliances that can be discriminated when using a haptic teleoperation system or the different amounts of damping, that can be rendered and perceived when using $\hookrightarrow$  VR interfaces and corresponding applications. An example is presented by Scilingo et al. that investigate the ability of a magnetorheologic haptic display to render compliances [49]. For the evaluation of haptic displays intended for communications, stimuli are constructed from the different attributes the system can display (for example speed, distance, frequency, amplitude etc.) and the information transfer is considered with respect to these attributes. This is illustrated in the following example.

Example: Tactile Shear Display by Gleeson et al. [ 18 ]

Gleeson et al. developed a tactile shear display to convey information in mobile applications. Figure 13.6 gives a picture of the completed system, that is based on a pin that can be moved in different directions with different kinematic properties (see [17, 23] for further information). A confusion matrix of several tests with the tactile shear interface, that displayed stimuli in North, South, East and West directions, is also given in Fig. 13.6.

Fig. 13.6

Evaluation result of a shear display (a) with regard to the displayed directions, used with permission by Brian T. Gleeson (b). Values in the confusion matrix are taken from [18] and report all stimuli presentations of the study without differentiating further experiment variables like moving distance and speed

Based on the values in the confusion matrix, the calculations according to Eqs. (13.1) and (13.2) give $IT_\text {est} = 1.23\,\text {bit}$ and $n_C = 2.35$. With regard to the above rules-of-thumb one could argue that the test setup contained a too little number of stimulus alternatives K to evaluate the maximum channel capacity, but that was not the intention of the original study the data where taken from. One has to note also, that Gleeson et al. conducted several studies with different hypothesis and the confusion matrix in Fig. 13.6 contains all trails from these different studies. It is only used as an example for the usage and interpretation of $\hookrightarrow$  IT here.

2.3 Information Input Capacity (Fitts’ Law)

The evaluation of Information Input Capacity is somewhat the opposite side of the assessment of information transfer as described in the last section. While this quantity measures the amount of information transferred from system to user (interaction path P’ in Fig. 2.24), the Information Input Capacity measures the amount of information that can be transferred from user to system (interaction path I’). The concept was developed by Fitts [16] and is often referred to as Fitts’ law. It describes the accuracy of movement with regard to the size of the movement. It was proven originally for unidirectional movement tasks like tapping (Fig. 13.7), peg-in-hole and item transfer and extended to two-dimensional taks by Accot and Zhai [2].

Fig. 13.7

Figure is based on [16]

Tapping experiment used to evaluate the information input capacity. Test persons were told to tap both target regions with width d at distance w from each other as fast as possible without missing the targets.

To measure the information input capacity, Fitts defined the index of performance $I_\text {p}$ according to Eq. (13.3)

$$\begin{aligned} I_\text {p}&= - \frac{1}{t} \log _2\left( \frac{w}{2d}\right) \text { in bits/s} \end{aligned}$$

(13.3)

with W as measure of the target size and A as distance between targets, as shown exemplary in Fig. 13.7. The logarithmic term is also called the index of difficulty $I_\text {D}$. To incorporate the temporal dimension the average time for a single movement t is used.

This approach is however difficult to use, when one wants to compare different systems or interaction setups. Therefore, the usage of the relation given in Eq. (13.4) is used more often. It relates the moving time $t_\text {m}$ with the Index of Difficulty $I_\text {D}$ and two device and test-dependent constants $c_a$ and $c_b$.

$$\begin{aligned} t_\text {m} = c_a + c_b \log _2\left( \frac{d}{w} +1 \right) \end{aligned}$$

(13.4)

Equation (13.4) can be derived from Eq. (13.3) directly, but contains a slightly modified formulation of the index of difficulty $I_\text {D}$ According to MacKenzie and Buxton this so-called Shannon formulation provides slightly better fits to given data, is more consistent with the underlying information theorem and ensures an always positive rating for the index of difficulty [37]. Given that, it is obvious, that the device and test dependent constant $c_b$ directly relates to the index of performance $I_\text {p}$. In an evaluation, one will conduct tests with different indices of difficulty of performance and record the movement time needed to move to and select a target. The values of $c_a$ and $c_b$ are determined by fitting equation (13.4) to the data. The such acquired indices of performance allow to measure the input capability of a user with a given system and the comparison of different haptic systems.

For haptics, one can find studies employing Fitts’ Law to investigate the effect of haptic feedback on task performance with different interface configurations [19, 46, 56] and multimodal interfaces [10].

3 User-Centered Evaluation Methods

The usage of haptic systems will not only have an effect on the task performance of a user, but also on the user himself. User-centered evaluation methods will give some insights into these processes in order to assess advantages and disadvantages of haptic systems in the investigated application.

Compared to task- and system-centered evaluation methods, there is no prevailing test form for user-centered methods. Depending on the intended informative value of the test one will find comparative tests as well as tests producing single test values.

3.1 Workload

Ergonomics and occupational sciences know different forms of workloads as defined by the standard ISO 10075. For the evaluation of haptic systems, the influence of a haptic system on the physical and mental workload of a user is probably the most interesting problem. For the evaluation of haptic systems, the assessment of workload is mainly of interest for the following intentions of haptic feedback:  

Lowering the workload:

If a haptic system is designed to lower the workload by adding haptic feedback to an existing application, one should verify if this goal is achieved. In that case, one will probably compare workload assessments in test conditions with and without haptic feedback. It might be possible to integrate this workload assessment in other evaluation tests like for example task-centered evaluations described above. In most of these cases, mental workload is of interest.

Assessment of new kinds of haptic interaction:

When a haptic interaction replaces other forms of interaction, it might be beneficial to evaluate the effects on physical (sometimes also mental) workload to gain knowledge about the usefulness of the new interaction scheme.

As an overview, Fig. 13.8 shows some influence factors on the workload of a human operator that probably can be influenced by the designer of a haptic system and the usage definition. Based on this considerations one should also note, that a medium workload is preferable for a safe usage of a haptic system—too low or too excessive demands on the user will result in errors.

Fig. 13.8

© Springer Nature, all rights reserved

Influences on the workload of a human user and possible outcomes. Figure based on [48]

In the following, a quite simplified view on workload evaluation is presented to have applicable methods for the comparison of different systems or different conditions of a single system. Aspects like emotional workload are neglected, since they mainly depend on the conditions a task is executed in and not on the system used [48]. The application of the methods presented here therefore does not fulfill all requirements of a workload analysis as formulated by ISO 10075-3.

One of the easiest ways to assess workload is the usage of standardized questionnaires. One of the widespread is the NASA Task Load Index (NASA-TLX) [20]. The resulting scale is a workload measure based on six different areas: mental demand, physical demand, temporal demand, performance, effort and frustration. Test persons first evaluate the amount of contribution of each of these areas to the overall workload, in a second step the areas are evaluated on bipolar scales [48]. Despite the NASA-TLX, there are quite a few of other workload assessment tools. An overview is given in [36], the German Federal Institute for Occupational Safety and Health (BAuA) also provides an online database and toolbox for this purpose [7].

Another possibility to assess workload is the analysis of biometric measures. Pulse rate, blood pressure, respiratory frequency, brain activity, $\hookrightarrow$  electromyography (EMG), pupil diameter, blinking frequency, skin conductivity and eye movement are indicators for a persons attention and concentration. Advantages of these biometric measures are the possibility of continuously data acquisition without major interference with the actual task. Disadvantages of these measures are the dependence on the individual user (therefore one often has to record a baseline before actual performing the experiment) and the lacking of a definite assignment of biometric measures to a defined measure of workload. Table 13.1 gives some biometric indications that can be used for a selection of suitable measurements.

Table 13.1 Biometric indications to newly arriving information and evidence for information processing. Table based on [48]

For haptic systems designed to assist a human operator for example in handling tasks with complex trajectories, the assessment of muscular activity and fatigue could be of further interest. While muscular activity is monitored by EMG, the assessment of muscular fatigue is somewhat more complicated. According to [55], EMG signals do not correlate well with muscular fatigue. The proposed evaluation based on maximum voluntary contraction (MVC) of the muscles is probably challenging to implement in a test of a haptic system. Further possibilities to assess muscular fatigue can be adapted from sport science literature [60], if the presented possibilities are not applicable or insufficient for the intended test.

3.2 Subjective Evaluation

While subjective evaluations can be done with regard to almost every aspect of the usage of a haptic system, the measure of the amount of immersion into a virtual or teleoperated environment is of interest for these kinds of applications. Pongrac reports several standardized questionnaires for this purpose [42] like for example the Witmer-Singer-Presence-Questionnaire or the ITC-Sense of Presence Inventory. More recently, Chertoff et al. presented an approach for the evaluation of multimodal virtual environments claiming to improve the integration of sensory, cognitive, affective, personal and social elements of experience [8]. Although primarily developed for the assessment of computer games, this could be a valid alternative for the evaluation of haptic telepresence systems and $\hookrightarrow$ VR interactions as well.

For the design of training systems, a subjective evaluation, i.e. a self-report of the test subjects, that compares the virtual or teleoperated training condition to the real condition without system-mediated haptic feedback is advisable to investigate acceptance processes of the users. Kron et al. investigate preferences of users for certain kinematic structures of a telepresence system for disposal of explosive ordances by using a subjective evaluation after usage [32]. Further evaluations could cover emotional and social aspects of a system. This may seem to be exaggerated for the majority of haptic systems, but is of importance in application areas like $\hookrightarrow$ Ambient Assisted Living (AAL) or other assistive systems for elderly and disabled users. In that case, one has also to consider impacts on other people not directly interacting with the haptic system that are only involved as a relative or assistant.

Self-reports can also be used to evaluate the subjective performance of users, when some kind of feedback about the task performance or criteria for good performance are given [59]. From the evaluation of usability aspects like the joy of use could be assessed by self-reports or even the assessment of verbal statements during the evaluation test [47, Chap. 6].

3.3 Learning Effects

Especially for training applications, the results of regular performance tests can be compared over time and learning effects can be quantified in terms of these test outcomes. Another approach to measure learning effects it the comparison of a trained group of subjects with an untrained control group in the intended application like for example shown by Ahlberg et al. for the effect of $\hookrightarrow$ VR training on the error rate in cholecystectomies by novice surgeons [4].

3.4 Effects on Performance in other Domains

Another approach to measure the effect of a haptic system on the user is to assess the performance in another domain. Predominantly one will find such kind of tests in areas, where a haptic system is intended to assist in a secondary task, while the primary task is not to be affected. Obviously, this is true for communication applications in vehicles. In that case, the standardized $\hookrightarrow$  Lane Change Test (LCT) is performed with interaction tasks using the haptic communication system as a secondary task to the driving on real roads or in a simulator, for example. Despite the standardized outcomes like the medium lane deviance, other parameters as for example the viewing direction and duration or the task performance of the secondary tasks can be evaluated. Examples for such kinds of evaluation of touch devices can be found in [13, 35, 50].

There are also cases, where such tests aim at the primary goal of the intended application: Várhelyi et al. investigate the effect of an active accelerator pedal that would create a counterforce when passing the speedlimit. The evaluation conducted in 284 vehicles showed an improvement of the driver’s compliance with the speed limit as well as reduced average speeds, speed variability, and emission volumes [54].

4 Formal Framework for Evaluation

According to ISO standard, there are three main aspects of assessment: Validation, Verification and Usability. They establish a link between the user, the requirements, and also the system under consideration. Validation establishes a relationship between the user and the requirements, verification relates to the requirements and the system, and usability is established between the user and the system (Fig. 13.9).

Fig. 13.9

Assessment main aspects and their relation

4.1 Validation

What is validation? It evaluates the specifications of the requirements in terms of their accuracy and completeness. It can be said that the answer to the question “Are we building the right system?” is validation [5]. This process performs a comparative assessment and shows whether the requirements definition is correct or not [26]. The important criterion should be identified from various sources and specified in the requirements. The usability test becomes more important than validation if the criterion cannot be determined.

Requirements reviews help validation alongside development. User requirements can be used as input to an acquisition process and validation can be done on them. Communication between users and developers has a great impact on the quality of validation. It plays an important role in the requirements documentation. Pre-validated haptic requirements can be found at ISO standards [27].

In tactile/haptic interaction, the validation is done by the interaction goal. So for this point, any requirement that is relevant to the goal is important.

For example, the goal could be to draw something by moving a pen in a virtual world. Detailed requirements can be made for the end effector gripped by the hand, the pressure of the pen on the virtual screen, the force feedback to the user and the simulation of colour and movement. From a technical point of view, it could be dynamic movement and pen pressure. Several scenarios can be used to provide the process. Each scenario can add further requirements such as pen speed, line size and thickness. Various limitations such as available technology and budget should also be considered.

To repeat the tests and get a wide range of results, several people can perform the tasks and get their opinions. Before or after this step, prioritisation of the requirements can be done and reviewed according to the ideas of different people. This can open up several useful objectives in the future development of tactile/haptic systems.

4.2 Verification

As with validation and requirements, verification checks the systems and their operations accuracy and completeness. Answering the question “Are we building the right system?” is validation [5]. It can be said that [26], verification checks whether the system can meet the design requirements or not.

To do this, various criteria need to be defined, and if there are no specific criteria, usability testing is a way to verify. By comparing system specifications and requirements, verification is done during the development process. However, verification during developement only compares user requirements with published system specifications, confident that the published specifications have been properly verified. The determination of appropriate measurement techniques has an impact on the quality of verification. Appropriate measures and measurement techniques can be found at ISO standards [27].

In general, electromechanics are the basis of tactile/haptic systems, regardless of the software for the tactile/haptic scenario used in the interaction. They are primarily focused on skin stimulation and use mainly mechanical stimulation. They also use chemical, thermal and electrical stimulation and are primarily focused on human body perception.

4.3 Usability

As mentioned earlier, usability is a link between the user and the system and tests the user’s ability to operate a system. Usability is also defined as the quality of system used by a user to achieve a particular goal with sufficient comfort, efficiency and accuracy [24]. It also includes the tasks, the equipment such as software and hardware, and the social environment of system use. In another hand, the answer to the question “is the system suitable for the users and their tasks?” is usability. Usability testing can be done both during development and acquisition of the system and in tactile/haptic systems, it can be defined on the basis of the quality of sensory transmission.

4.4 Handling Requirements for Devices

As mentioned before, each tactile/haptic device should have a set of specifications (according to the appropriate ISO standard) that relate to its performance and intended use. Based on these specifications, a meaningful comparison between different tactile/haptic devices is possible and the same measurement methods should also be used. Each requirement should therefore be measurable and testable.

4.4.1 What Should Be Measured?

If a designer wants to fully characterize the device, each clause in the specification should be either noted or measured, depending on what quantity is to be measured. These measurable tactile/haptic specifications are divided into several categories, such as environment, force, general and temporal, and ergonomics.

Environment: temperature, texture, weight, shape, electrical, mechanical and acoustic properties, and installation and maintenance.

Force: maximum and minimum continuous force and torque, peak acceleration, force and torque, inertia and resistance.

General and temporal: system delay, bandwidth, frequency, adjustability.

Ergonomic: interface, motion parameters, workspace, degrees of freedom.

4.4.2 Measurement Resolution

There are different ways to measure the characteristics of the tactile/haptic system. These methods are defined depending on the feature. However, as a rule, the resolution of the measuring devices should be about ten times higher than the measurement resolution expected by the user. In addition, high-resolution measurements are not always necessary. It may be sufficient to measure a working area to the centimeter rather than to the millimeter.

4.4.3 How should Be Measured

A test environment should be measurable and repeatable. The procedure is designed according to the situation. The procedure should include at least three usability components specified by ISO 9241-920 [27]: Effectiveness, Efficiency, and Satisfaction. For usability evaluation, different users put themselves in the test situation and follow the test procedure using the tactile/haptic system [15].

4.4.4 Effectiveness

The useful tests will provide an effectiveness and success score. This score will be determined based on the type of test. For example, if the tests are yes/no tests, the score will be the ratio between successful and total attempts. The more users, the lower the indicative bias. Also, the selection of users should be random and the accuracy of results depends on the desired level of certainty required.

The identification and reading speed, latency, percentage of success and target achievement, and average accuracy can be used to measure effectiveness.

4.4.5 Efficiency

Target attainment time in a typical tactile/haptic interaction can be measured. Thus, the ratio of the individual scores for determining the effectiveness divided by the time required can yield efficiency. The average of these scores is the baseline efficiency. It is also more useful to decide which of two tests is more efficient. Efficiency can include: Time to complete the first task, mean time to perform a task, and time to correct errors.

4.5 Satisfaction (ISO)

ISO 16982 [24] presented twelve methods for evaluating user satisfaction with an interaction. Since these methods cover a wide range, many factors should be considered to select one of them. These factors can be the simplicity or complexity of the task, the type of object being acquired, the design or operation of a system, and also the form of user participation.

For example, if you are using a questionnaire, the questions may be completed by the user and based on an interview. Also, an ordinal scale should be used to assess user satisfaction. For example, if the question is “The procedure was easy to understand,” the answers can range from “completely agree” to “completely disagree.”

The analyst collect data on usability components and specific interaction aspects. These data were to be analyzed using common statistical methods to obtain the results. ISO 25062 [25] shows usability test reports, which are an industry standard format.

For example, in a virtual glove and touchscreen test, a monitoring computer measures the success of communication and the time required. A questionnaire also provides information about the user experience. In practice, comparing two or more means of interaction provides higher quality data. By changing a small number of variables, the effect of the variables can be compared and the most effective parameters can be inferred. In this example, these parameters may be glove vibration, internet speed, and frequency.

5 Evaluation of Haptic Systems, Industial Standards

• Wenliang Zhou &
• Jörg Reisinger 

The physical evaluation of haptic displays due to their ability of representing a specific haptic property is mandatory when developing haptic devices for high volume-series production devices in mobile, white goods or automotive market. The grown awareness of subjective quality and the request for increased haptic effect variety requires a clear and transferable objective description of the haptic feeling to describe, define, reproduce and evaluate it on any platform. This subsection shortly discusses the current state of haptic measurement methodology focusing on the push-feedback.

5.1 Evaluation of Control’s Subjective Haptic Feedback

To specify and evaluate their haptic characteristics, static measurements i.e. force-displacement curves for push buttons and torque-angle curves for rotary controllers, are frequently used in industry since the last three or four decades [6, 31, 41, 44, 57]. However, this measurement only describes the static properties of control elements and that why, cannot capture dynamic effects. For example, control elements, which have the same static properties, but still feeling differently, the static measurement method is not able to specify.

However, a number of results indicate that also dynamic effects play a role in haptic perception. For example [43] describes that inertia and stiffness have an influence on the perception of damping, as well as [34] is showing the mutual influence of damping on the perception of inertia. [33] describes the significantly increased realism of event-based feedback of virtual surfaces and [44] describes event-based-perception observed on push button perception (also compare Chap. 5 Haptic Design of Mechanical Controls). At the same time, the practical application of static measurement reaches its limits as well, not being able to describe certain cases or new applications. In summary, it is necessary to record and evaluate the dynamic properties of haptic feedback in order to describe the perceived quality of haptic events.

At the beginning of this task, it is worthwhile looking at the entire system and the relevant properties of the whole test procedure. The results should generally be transferable and rate subjective properties by objective values. In order to analyze these effects of the dynamic behavior of control elements on the haptic perception and to specify the desired dynamic behavior for their design, we need to be able to characterize the interaction that is taking place between control elements and the human actuating them. [61] introduced a haptic measurement method that measures the dynamic interaction between a control element and an instrumented robotic probe. Since the resulting interaction is not only depending on the control element, but also on the mechanical properties of the probe. It requires the use of a finger-like measurement probe instead of a rigid probe to imitate human-like interactions. Using this new interaction-based dynamic measurement method is capturing dynamic haptic characteristics of control elements in the real working point very efficiently. The measurement of an active haptic system by for example a laser vibrometer is limited to a purely technical description of the system and its Eigen-vibrations, without the transferable properties to perceived haptics, due not considering the influence of the human finger.

5.1.1 Static Measurement

A common method to measure force-displacement curves is using a positioning system with a rigid probe to push the control element with a constant velocity [6, 31, 41, 44, 57], as shown in Fig. 13.10(left). The pushing velocity has to be chosen low (around 0.1 mm/s) so that dynamic effects of control elements due to inertia or damping are negligible and only stationary behavior is measured. From the force-displacement curve, technical features like lead travel and switching force (Fig. 13.10(right)) are derived and correlated with subjective percepts by psychophysical studies, already showing up the limits in [44] as also described in Sect.14.1. Note that this curve describes only static properties e.g. the stiffness of control elements.

Fig. 13.10

Static measurement for a push button: Measurement device (left); Force-displacement curve and technical features derived from the curve (right)

However, when pushing a button or turning a rotary controller, the interaction between the finger and the control element is a dynamic process and control elements with the same stiffness, but different inertias or viscoelasticity can still feel very differently, although all the static measurements are the same (Fig. 13.10 for an example). Therefore, static measurements are not sufficient to capture all haptic characteristics of control elements (Fig. 13.11).

Fig. 13.11

Two same push buttons (left), one of them with an additional mass (2.5 g), feel differently (the snap of the one with more mass feels dull and unclear), but their static measurements (right) are the same

Furthermore, over the past decade, control elements have no longer been limited to push buttons and rotary controllers. Diverse new concepts have been developed, for example, touchpads or touchscreens with active haptic feedback (Fig. 13.12(left)). The stiffness of such touchpads can be described by a progressive spring, which means that there appears no snap¹ in the static force-displacement curve. The actual snap caused by an impulse generated by the actuator of the touchpad to realize the sensation of a switch, does not change the static force. If this snap is generated differently, the touchpad will feel differently, although its stiffness is the same (Fig. 13.12(right) for an example). Again, the static measurement cannot capture these dynamic haptic characteristics.

Fig. 13.12

Touchpad with active haptic feedback (left); Two touchpad samples with different haptic characteristics generated by their actuators, feel differently (one snap feels strong and the other feels weak), but they all have the same static force-displacement curve. In the curve, no snap (a drop in force) can be found but only a small spike (right)

To measure the dynamic properties of control elements, one might consider using the static measurement device to excite the control element and then identify its dynamic parameters. However, to sufficiently stimulating the control element, the input signal must be highly dynamic and, at the same time the total travel of a control element is usually very short, which means that a highly dynamic measurement device with a very high sampling frequency is necessary. More importantly, there are diverse types of control elements in an automobile, and it is difficult to find a general model structure to describe all of them. At all, it is necessary to find an appropriate model structure for each specific control element type, before identifying them. Summarizing, direct identification of control element parameters is not a cost- and time-effective approach to capture dynamic characteristics of control elements in automotive industry.

5.1.2 Interaction-Based Dynamic Measurement

To overcome the restrictions of static measurements, a new interaction-based dynamic measurement method (IDM) is introduced. The main idea of this method is that instead of directly identifying dynamic parameters of control elements, characterizing the element indirectly by features derived from interaction signals. Considering the human finger as a mechanical system, exciting this system by the interaction force applied on the finger. By measuring this force along with the displacement and its derivatives, indirectly capturing the dynamic haptic characteristics of control elements. The interaction depends not only on the control element but also on the finger as they are two physically coupled dynamical systems and the dynamics of one affects the overall dynamics. Thus, it is desirable that the measurement device mimics the dynamic behavior of an interacting human finger to imitate human-like interactions. If the imitated interaction signals differ significantly from the real interaction signals, some information relevant to haptic perception might have been filtering out or some irrelevant information might have been scaling up.

The main principle of the new measurement method is shown in Fig. 13.13(left), the measurement probe and the control element are modeled as two coupled mass-damper-spring systems. The impedance parameters $c_p$, $d_p$, $m_p$ and $c_s(x)$, $d_s$, $m_s$ represent the stiffness, damping and mass of the probe and of the push button respectively (subscript p for probe and s for switch). Note that the static measurement only is able to measure $c_s(x)$ which is non-linear and depends on the displacement x(t) . The input of the coupled system is the probe movement $x_p(t)$. The interaction force is denoted by f(t) . The equation of the motion of this system given by:

$$\begin{aligned} m_p \ddot{x}(t) + d_p (\dot{x}(t)-\dot{x}_p(t)) + c_p (x(t)-x_p(t))= & {} f(t) \end{aligned}$$

(13.5)

$$\begin{aligned} m_s \ddot{x}(t) + d_s \dot{x}(t) + c_s(x) x(t)= & {} f(t). \end{aligned}$$

(13.6)

Fig. 13.13

Interaction model when pushing a button (left); Finger-like measurement device (right)

Measurement Device and Finger Parameters

Instead of the rigid probe used in static measurement, a mass-damper-spring system is chosen as a finger-like probe to push buttons with a constant finger-like pushing velocity.² Choosing the components of the probe in an iterative way and testing in each iteration whether the probe can reproduce the real interaction is necessary. The finally achieved probe is shown in Fig. 13.13(right). It consists of a spring element with a certain damping and an accelerometer. The impedance parameters of this probe are 1.8 N/mm (stiffness), 0.92 Ns/m (damping) and 6.6 g (mass). The reason of using an accelerometer instead of a force sensor is that a force sensor is usually too big and too heavy compared with a finger. Note that in Eq. (13.5), the interaction force f(t) can be derived from acceleration $\ddot{x}(t)$ using integration (the starting conditions must be known though), when the movement and impedance of the probe are known.

This probe is validated by measuring and comparing the real interaction between fingers and control elements and the reproduced interaction between the probe and the same control elements. The probe with an accelerometer pushes different control elements at 7 mm/s and the acceleration during the snap is measured. Using the same accelerometer, attached on the surfaces of the control elements, fingers of different subjects push the control elements with a similar gesture and velocity as typically used when pushing buttons in an automobile. The results show that the probe is a finger-like measurement device as it can reproduce similar interactions as a finger. One of those results is presented in Fig. 13.14.

Fig. 13.14

Three Measurements of acceleration (left, in time domain; right, in frequency domain) between three different fingers and a push button (red curves) are compared to one measurement of acceleration between the probe and the same button (blue curve). The two groups of the curves are very similar

Some simulation examples in Fig. 13.15 is demonstrating how different the interaction will be if the mechanical parameters of the probe are changed.

Fig. 13.15

Acceleration in the interaction during pushing a button is simulated (Interaction I). If the probe parameters are changed in the simulation, the interaction will be changed significantly. Interaction II: Probe damping is 3 times greater; Interaction III: Probe stiffness is 10 times greater; Interaction IV: Probe mass is 10 times greater

Application Examples

In order to show the effectiveness of the new measurement method, acceleration signals during a snap of the push buttons and the touchpads mentioned in Sect. 13.2, Figs. 13.10 and 13.12, are measured by the new finger-like probe pushing with a velocity of 7 mm/s.

Capturing the different haptic properties of the button with the additional mass by the dynamic measurement method. As shown in Fig. 13.16, this button has smaller acceleration as large amounts of the signal energy is filtered out due to the mass. Thus, the snap feels dull and unclear.

Fig. 13.16

Dynamic measurement of two push buttons with the same force-displacement curve, but different masses

The two touchpads with the same stiffness, but different active feedback differs significantly in the dynamic measurement, which is shown in Fig. 13.17. The sample I has a much larger acceleration and more signal energy and therefore, its feedback feels stronger.

Fig. 13.17

Dynamic measurement of two touchpad samples with the same stiffness, but different active haptic feedback

Derivation of Features from Measurement

Some technical features are derived from the dynamic measurement signals e.g. the measured acceleration, as shown in Figs. 13.14 and 13.15, the acceleration signal shows a damped oscillation. In order to describe this oscillation, the following features are defined (Fig. 13.18):

• Max. and min. value of the acceleration signal $a_{max,min}$.

• Duration of the interaction $T_d$ starting with the first and ending with the last time when the acceleration reaches 10% of its maximum.

• Energy of the acceleration signal E (defined by the sum of all squared acceleration values).

• Frequency range $f_{start}$, $f_{end}$ described by the width and the location of the frequency range, in which the acceleration covers 75% of the area spanned by the acceleration from 0 Hz to 1.5 kHz.³ The dominant frequency $f_d$ denotes the location of the peak of the spectrum.

• Ratio of peak value to root mean squared value of the acceleration in frequency domain p2r: This feature roughly describes the form of the amplitude spectrum of the acceleration, usually applied as a feature from acoustic measurement.

Fig. 13.18

Technical features derived from the dynamic measurement

Based on the evaluation of large amounts of touchpads of Mercedes-Benz C Class 2014 by a group of experts deriving correlation between these features and human perception. For example a “nice feeling” of a push button or a touchpad requires large values of $a_{max,min}$ and small $T_d$ at the same time. Additionally, the acceleration should be distributed in the range around 300 Hz. Using this preliminary knowledge improving the haptic characteristics of touchpads that the haptic feedback is not feeling artificial, but similar sharp and clear like a mechanical push button. Besides these subjective perceptional results, the intensive use in the end-of-line measurement approved the robustness of the measurement system.

Examples of IDM devices

In addition to the initial system, in series production development of several EOL testing devices took place. The measurement systems show comparatively high test-speed and the concept has been proven to be mechanically robust. The measuring equipment capability of the IDM moves at a very good level, i.e. the repeatability of measurements is very good within one measurement system. At all, the replacement of measurement fingers or the comparability of results between different measurement systems causes some difficulties. Due to the nonlinear behavior of the mathematical calculation of the features as well as their depending tolerances is limiting the possibility of a mathematical correction of the values. Therefore, deviations of the finger-impedances show significant influence on the feature values so that a comparison in high production volumes is not sufficiently valid. That why, an adjustment and evaluation of the finger impedance regarding standard values is necessary to reach comparability of the different systems’ measurements.

The EOL test bench of the EQS’ Hyperscreen is shown in Sect. 14.1. Besides active haptic systems, Mercedes-Benz is also testing classic mechanical push buttons via IDM.

Off the Shelf Systems

The Force Feedback module from PANOVO tec GmbH [53] covers the entire range from development to EOL testing and already launches the second more compact generation. Adjusting the measuring finger before delivery and automatically calibration during operation to evaluate finger impedance reaches sufficient comparability of the results. An acoustically optimized drive and a stabilization of the finger during positioning against lateral forces by an adjustable preload are enabling the device for robot-based end-of-line testing. Figure 13.19 shows the test finger F121 used for laboratory operation as well as automated in EOL. Customization of the algorithms of the evaluation software is possible as well.

Fig. 13.19

Panovotecs’ IDM probe F121

Grewus GmbH [53], supplier for acoustic and haptic actuators, presented on the “automotive interiors expo 2021” fair a cost-effective system for dynamic haptic testing, called ArFi. Its focus lies on comparative measurements during the development phase in the laboratory. Since performing no special adjustment or calibration, the comparing of the results is possible only within a single device. Displaying time and frequency domain behavior as well as an intensity value called GHIV (Grewus Haptic Intensity Value). Placing a second accelerometer at any position of the component allows identifying disturbing vibrations. A microphone allows recording the sound and noise of the component synchronously.

The Syntouch Finger [1] has its focus on surface haptic properties. At all, it could also be an interesting approach for the evaluation of haptic events, but the use of rubber may be not the ideal material for long-term end-of-line measuring equipment capability.

5.2 Discussion and Outlook

Any engineering work focuses on designing a product. A successful industrialization of haptic actuation technology in any industrial context however requires always two things:

• a technology with high performance and optimized quality to cost ratio for the intended market

• a measure for this performance and quality

Both items are not to be underestimated. A successful product is nothing without a corresponding quality control. Especially in the area of haptic technology this is something which always needs to be developed according to the application at hand. The Mercedes C-class touchpad is an excellent example of this conjunction of research and development and is proposed an example to extend the real product range of active haptic devices available to the market.

6 Conclusion

As it can be seen from this section, the evaluation of haptic systems is complex and exhibits a large number of different facets. For each newly developed task-specific haptic device, evaluation methods and goals have to be selected from the above mentioned (and other applicable) measures. Depending on the kind of application, existing studies can give hints about the selection of applicable methods. The works of Wildenbeest et al. evaluating teleoperated assembly tasks and the evaluation of an assistive system for minimal invasive surgery by McMahan et al. [38] should be recommended here because of the wide scope and the thorough methodology for this kind of systems. For new kinds of universal haptic interfaces, the work of Samur is naturally a must-read [46]. At the same time, industrial companies step more and more into the area of haptic measurement technology. The future is promising for better and more objective evaluations accessible to a broad range of applicants.

Recommended Background Reading

• [21]] Hayward, V. & Astley, O. R.: Performance Measures for Haptic Interfaces In: Robotics Research, 1996.

  Extensive list of possible physical measures for the evaluation of haptic interfaces

• [46]] E. Samur: Performance Metrics for Haptic Interfaces. Springer, 2012.

  The probably most advanced work about evaluation techniques for haptic interfaces with a strong focus on the interaction with virtual environments.