Keywords

1 Introduction

Virtual Reality (VR) experiences are used by an increasing number of people, through the introduction of devices that are more accessible to the market. Many VR applications have been launched and are becoming parts of our daily life, such as simulators and games. To deliver a highly immersive VR experience, a significant number of senses have to be stimulated simultaneously according to the activity that the users perform in the VR environment.

The tactile information from haptic interfaces improves the user’s perception of virtual objects. Haptic devices introduced in [1,2,3,4,5], provide haptic feedback at the fingertips and increase the immersion experience in VR.

Fig. 1.
figure 1

a) Sensor array at the user’s palm to record the tactile perception of objects. Fifteen FSRs were located according to the physiology of the hand, using the points over the joints of the bones, and the location of the pollicis and digiti minimi muscles. b) Experimental Setup. Each object was placed on the top of the FT300 force sensor to measure the applied normal force. The data from the fifteen FSRs and the force sensor are visualized in real-time in a GUI and recorded for future analysis.

Full size image

Many operations with the hands involve more that one contact point between the user’s fingers, palm, and the object, e.g., grasping, detection, and manipulation of objects. To improve the immersion experience and to keep the natural interaction, the use of multi-contact interactive points has to be implemented [6, 7]. Choi et al. [8, 9] introduced devices that deliver the sensation of weight and grasping of objects in VR successfully using multi-contact stimulation. Nevertheless, the proposed haptic display provides stimuli only at the fingers and not on the palm.

The palm of the users plays an essential role in the manipulation and detection of objects. The force provided by the objects to the user’s palms determines the contact, weight, shape, and orientation of the object. At the same time, the displacement of the objects on the palm can be perceived by the slippage produced by the forces in different directions.

A significant amount of Rapidly Adapting (RA) tactile receptors are present on the glabrous skin of the hand, a total of 17, 023 units on average [10]. The density of the receptors located in the fingertips is \(141\ \mathrm{units/cm}^2 \), which is bigger than the density in the palm \(25\ \mathrm{units/cm}^2\). However, the overall receptor number on the palm is compensated by its large area, having the \(30\%\) of all the RA receptors located at the hand glabrous skin. To arrive at this quantity, we should cover the surface of the five fingertips. For this reason, it is imperative to take advantage of the palm and to develop devices to stimulate the most significant area with multi-contact points and multi-modal stimuli. Son et al. [11] introduced a haptic device that provides haptic feedback to the thumb, the middle finger, the index finger, and on the palm. This multi-contact device provides kinesthetic (to the fingers) and tactile stimuli (at the palm) to improve the haptic perception of a large virtual object.

The real object perception using haptic devices depends on the mechanical configuration of the devices, the contact point location on the user’s hands, and the correctness of the information delivered. To deliver correct information by the system introduced by Pacchierotti et al. [12], their haptic display was calibrated using a BioTac device.

There are some studies on affective haptics engaging the palm area. In [13], the pressure distribution on the human back generated by the palms of the partner during hugging was analyzed. Toet et al. [14] studied the palm area that has to be stimulated during the physical experience of holding hands. They presented the areas of the hands that are stimulated during the hand holding in diverse situations. However, the patterns extraction method is not defined, and the results are only for parent-child hand-holding conditions.

Son et al. [15] presented a set of patterns that represent the interaction of the human hand with five objects. Nevertheless, the investigated contact point distribution is limited to the points available in their device, and the different sizes of the human hands and the force in the interaction are not considered.

In the present work, we study the tactile engagement of the user’s palms when they are interacting with large surfaces. The objective of this work is to introduce a method to investigate the sensation on the user’s palm during the interaction with objects, finding the area of interaction, and the normal force delivered. This information is used to the reproduction of tactile interaction, and the development of multi-contact wearable tactile displays. The relation between the applied normal force and the location of the contact area should be found, considering the deformation of the palm interacting with different surfaces.

2 Data Acquisition Experiment

A sensor array was developed to investigate the palm area that is engaged during the interaction with the different surfaces. Fifteen FSRs were located according to the physiology of the hand, using the points over the bones’ joints, and the location of the pollicis and digiti minimi muscles. In Fig. 1 the distribution of the fifteen FSRs is shown.

The fifteen FSRs are held by a transparent adhesive contact paper that is attached to the user’s skin. The transparent adhesive paper is flexible enough to allow users to open and close their hands freely. The different shape of the user’s hands is considered every time the array is used by a different user. The method to attach the FSRs on the user’s palms is the following: a square of transparent adhesive paper is attached to the skin of the users, the points, where the FSR must be placed, are indicated with a permanent marker. After that, the transparent adhesive paper is taken off from the hand to locate the fifteen FSRs at the marked points. The FSRs are connected to ESP8266 microcontroller through a multiplexer. The microcontroller receives the data from the FSRs and sends it to the computer by serial communication.

The hand deformation caused by the applied force to the objects is measured in this study. The force applied to the objects is detected by a Robotiq 6 DOF force/torque sensor FT300. This sensor was chosen because of its frequency of 100 Hz for data output and the low noise signal of \(0.1\ N\) in \(F_z\), which allowed getting enough data for the purposes of this study. The sensor was fixed to a massive and stiff table (Siegmund Professional S4 welding table) using an acrylic base. An object holder was designed to mount the objects on top of the force sensor FT300.

2.1 Experimental Procedure

Four surfaces with different diameters were selected, three of them are balls, and one is a flat surface. The different diameters are used to focus on the position of the hand: if the diameter of the surface increases, the palm is more open. The diameters of the balls are 65 mm, 130 mm, and 240 mm.

Participants: Ten participants volunteering completed the tests, four women and six men, aged from 21 to 30 years. None of them reported any deficiencies in sensorimotor function, and all of them were right-handed. The participants signed informed consent forms.

Experimental Setup: Each object was placed on the top of the FT300 force sensor to measure the applied normal force. The data from the fifteen FSRs and the force sensor are visualized in real-time in a graphical user interface (GUI) and recorded for future analysis.

Method: We have measured the size of the participant’s hands to create a sensor array according to custom hand size. The participants were asked to wear the sensors array on the palm and to interact with the objects. Subjects were asked to press objects in the normal direction of the sensor five times, increasing the force gradually up to the biggest force they can provide. After five repetitions, the object was changed, and the force sensor was re-calibrated.

2.2 Experimental Results

The force from each of the fifteen FSRs and from the force sensor FT300 was recorded at a rate of 15 Hz. To show the results, the data from the FSRs were analyzed when the normal force was 10 N, 20 N, 30 N, and 40 N. In every force and surface, the average values of each FSR were calculated. The average sensor values corresponding to the normal force for each surface are presented in Fig. 2.

Fig. 2.
figure 2

Number and location of contact points engaged during the palm-object interaction. The data from the FSRs were analyzed according to the normal force in each surface. The average values of each FSR were calculated. The active points are represented by a scale from white to red. The maximum recorded value in each surface was used as normalization factor, thus, the maximum recorded value represents \(100\%\). Rows A, B, C, D represent results for a small-size ball, a medium-size ball, a large-size ball, a flat surface, respectively. (Color figure online)

Full size image

From Fig. 2 we can derive that the number of contact points is proportional to the applied force. The maximum number of sensors is activated when 40 N is applied to the medium-sized ball, and the minimum number is in the case when 10 N is applied to large-size and flat surface.

The surface dimension is playing an important role in the activation of FSRs. It can be observed that the surfaces that join up better to the position of the hand activate more points. The shape of the palm is related to the applied normal force and to the object surface. When a normal for of 10 N is applied to a small-size ball, the number of contact points is almost twice the contact points of other surfaces, it increases only to seven at 40 N. The number of contact points with the large-size ball at 10 N is two, and increases to eight at 40 N. Instead, the flat surface does not activate the points at the center of the palm.

3 User’s Perception Experiment

To verify if the data obtained in the last experiment can be used to render haptic feedback in other applications, an array of fifteen vibromotors was developed. Each of the vibromotors was located in the same positions as the FRS, as shown in Fig. 1. The vibration intensity was delivered in correlation with the acquired data from the Sect. 2 to discriminate the applied force from the hand to the large-size ball (object C).

3.1 Experimental Procedure

The results from the big ball were selected for the design of 4 patterns (Fig. 3). The patterns simulate the palm-ball interaction from 0.0 N to 10 N, 20 N, 30 N and 40 N, respectively. The pattern was delivered in 3 steps during 0.5 s each one. For instance in the 0.0 N - 10 N interaction, firstly the FRS values at 3 N were mapped to an vibration intensity and delivered to each vibromotor during 0.5 s, then similar process was done for the FRS values at 6 N and 10 N.

Fig. 3.
figure 3

Tactile presentation of the patterns with the array of vibration motors a) the arrangement of fifteen vibromotors is shown. b) P1, P2, P3, and P4 represent the actuated vibromotors on the palm and the intencity of vibration of each actuator (by red gradient color). (Color figure online)

Full size image

Participants: Seven participants volunteering completed the tests, two women and five men, aged from 23 to 30 years. None of them reported any deficiencies in sensorimotor function, and all of them were right-handed. The participants signed informed consent forms.

Experimental Setup: The user was asked to sit in front of a desk and to wear the array of fifteen vibromotors on the right palm. One application was developed in Python where the four patterns were delivered and the answers of the users were recorded for the future analysis.

Method: We have measured the size of the participant’s hands to create a vibromotors array according to the custom hand size. Before the experiment, a training session was conducted where each of the patterns was delivered three times. Each pattern was delivered on their palm five times in random order. After the delivery of each pattern, subject was asked to specify the number that corresponds to the delivered pattern. A table with the patterns and the corresponding numbers was provided for the experiment.

3.2 Experimental Results

The results of the patterns experiment are summarized in a confusion matrix (see Table 1).

Table 1. Confusion matrix for the pattern recognition.
Full size table

The perception of the patterns was analyzed using one-factor ANOVA without replication with a chosen significance level of \(\alpha <0.05\). The \(p-value\) obtained in the ANOVA is equal to 0.0433, in addition, the Fcritic value is equal to 3.0088, and the F value is of 3.1538. With these results, we can confirm that statistic significance difference exists between the recognized patterns. The paired t-tests showed statistically significant differences between the pattern 1 and pattern 2 (\(p=0.0488 < 0.05\)), and between pattern 1 and pattern 3 (\(p=0.0082 < 0.05\)). The overall recognition rate is 84%, which means that the user can distinguish four patterns with high confidence.

4 Conclusions and Future Work

The sensations of the users were analyzed to design robust multi-contact tactile displays. An array of fifteen Interlink Electronics FSRTM 400 Force Sensitive Resistors (FSRs) was developed for the user’s palm, and a force sensor FT300 was used to detect the normal force applied by the users to the surfaces. Using the designed FSRs array, the interaction between four convex surfaces with different diameters and the hand was analyzed in a discrete range of normal forces. It was observed that the hand undergoes a deformation by the normal force. The experimental results revealed the active areas at the palm during the interaction with each of the surfaces at different forces. This information leads to the optimal contact point location in the design of a multi-contact wearable tactile and haptic display to achieve a highly immersive experience in VR. The experiment on the tactile pattern detection reveled a high recognition rate of 84.29%.

In the future, we are planing to run a new human subject study to validate the results of this work by a psychophysics experiment. Moreover, we will increase the collected data sensing new objects. With the new dataset and our device, we can design an algorithm capable to predict the contact points of an unknown object. The obtained results can be applied in the development of multi-contact wearable tactile and haptic displays for the palm.

The result of the present work can be implemented to telexistence technology. The array of FSR senses objects and the interactive points are rendered by a haptic display to a second user. The same approach can be used for affective haptics.