Keywords

1 Introduction

The essential role of tactile devices is to deliver information, and the information is often abstract and categorical [2, 6]. Research has sought effective methods in improving the amount of information that can be transferred by tactile communication. In this paper, we report a preliminary study that quantifies the extent to which the information transmission capability is preserved when two different types of tactile stimuli, vibrotactile and thermal, are factorially combined into multimodal stimuli.

A large number of previous studies measured the information transfer (IT) for vibrotactile stimuli. When a single vibration actuator is used, the highest IT value reported is 3.06 bits [17]. When multiple vibration actuators are used, the highest IT value reported is 7.02 bits [7]. In comparison, the IT values reported for thermal stimuli are rare. The highest IT value reported for the identification of thermal patterns is 2.13 bits [11]. Furthermore, there have been attempts to test the IT for multimodal stimuli. Shim et al. [9] measured the IT values for the simultaneous and sequential patterns of vibration and wind. The IT values were 1.70 bits (simultaneous) and 3.29 bits (sequential), where the IT values for individual vibration and wind were 1.76 bits and 1.23 bits, respectively. A recent comprehensive review on the IT of tactile displays is available in [13].

Presenting vibrotactile and thermal stimuli simultaneously is a promising approach to enriching tactile interaction [14, 16]. If the two tactile channels are sufficiently independent, vibrotactile-thermal stimuli may offer a substantially greater information transmission capacity (ITC). This research hypothesis was investigated by two perceptual experiments. In Exp. 1, we measured the individual ITCs of the vibrotactile and thermal channels. Only frequency was varied for the vibrotactile stimuli, and the number of stimuli was increased from 3 to 7. For the thermal stimuli, temperature change direction and rate were controlled. The number of stimuli was between 5 and 9. In Exp. 2, we factorially combined the optimum number of vibrotactile and thermal stimuli to make multimodal stimuli and obtained their estimated IT values (\(IT_{est}\)). Then, we examined whether the unimodal \(IT_{est}\)s and the multimodal \(IT_{est}\) satisfy the additivity, which means the complete perceptual independence of the two channels.

Our study can contribute to 1) the decision of optimal vibrotactile or thermal stimuli for user interaction purposes, 2) the design of vibrotactile-thermal stimuli that are highly recognizable for information and communication purposes.

2 Exp. 1: Unimodal Stimuli

We estimated the ITCs for both vibrotactile and thermal stimuli. The experiments reported in this paper were approved by the Institutional Review Board at POSTECH (PIRB-2021-E054).

Fig. 1.
figure 1

Experiment setup (left) and apparatus for stimulation (right).

Full size image

2.1 Methods

Apparatus. We implemented a device for vibrotactile and thermal stimulation (Fig. 1). Its vibrotactile module consisted of a mini-shaker (Bruel & Kjær, 4810) and a power amplifier (Bruel & Kjær, 2718). The thermal module included a Peltier module (20 \(\times \) 20 \(\times \) 3.1 mm; MULTICOMP, MCPE1-07106NC-S), three DC motor drivers, a thermistor (SEMITEC, 223Fu3122-07U015), a microcontroller (Arduino Due), and a water-cooler. The temperature change rate of the Peltier module was controlled within ±4 ℃/s (0.1 ℃ average error) by PID control. The thermal module was put on the vibrotactile module using an acrylic connector. To block vibration propagation from the water-cooler, a silicon damper attenuating noise over 95% was placed between the tactile module and the cooler.

Fig. 2.
figure 2

Description of vibrotactile and thermal stimuli.

Full size image

Experimental Conditions. Figure 2 depicts the time profiles of vibrotactile and thermal stimuli. We designed three conditions for vibrotactile stimuli with the number of stimuli \(N=\) 3, 5, and 7 (see Table 1). The frequencies of the vibrotactile stimuli were varied from 40 400 Hz in equidistant intervals on a logarithmic scale, stimulating RA 1 and PC channels [1]. The adjacent frequency differences were larger than the vibrotactile Just Noticeable Difference (JND) (approximately 18% [8]). The intensity of each stimulus was scaled to be 33 dB from the detection threshold on the finger at its frequency [3]. All of the vibration stimuli had the same duration of 1.5 s.

We also designed three conditions for thermal stimuli. Their baseline temperature was 32 ℃. Considering the heat pain threshold, we varied the temperature change rate within ±4 ℃/s [15]. Given the number of stimuli \(N=\) 5, 7, and 9, the change rates were evenly spaced in a linear scale, as shown in Table 1. The temperature changes from the baseline were higher than the difference thresholds for both warm and cool stimuli (+0.23 and −0.14 ℃ at 33 ℃ [12]).

The thermal stimuli had three phases. In the beginning, the temperature changed to the target temperature at a given change rate over 1.5 s. In the second phase, the temperature remained for 0.5 s for stability. Then, the temperature returned slowly to the baseline temperature over 8 s.

Table 1. Experimental conditions of Exp. 1
Full size table

Participants. The experiment had a between-subjects design. Each experimental condition was tested with seven participants each (see Table 1 for their demographics). No participants participated in the conditions of the same stimulus type. Before the experiment, the participants were informed of the experiment’s goals and procedure via a written document, and then they signed a consent form. They were paid at the rate of KRW 10,000 (\(\approx \) USD 9) per hour.

Procedure. Participants put their right index fingertip on the Peltier module. Note that vibration stimuli were presented through the Peltier module (Fig. 1). They manipulated a experiment program using their left hand. They wore noise-canceling headphones which played white noise to block environmental sounds.

The thermal conditions had additional constraints: 1) The room temperature was controlled to 25 ℃; 2) The Peltier module maintained its temperature to 32 ℃ when there was no stimulus; 3) Participants were asked to put their fingertips on the Peltier module for 2 s before stimulus onset to make their contact skin temperature to 32 ℃; and 4) They were also asked to take off their fingers from the Peltier module after receiving 1.5 s of thermal stimulus.

To complete each experimental condition, participants went through five sessions: training 1, practice 1, training 2, practice 2, and main session. In the training session, participants experienced the haptic stimuli assigned to the condition and wrote down their identification criteria on a sheet of paper for at least 5 min. In the practice session, participants perceived each stimulus and were asked to find the correct stimulus. Correct answer feedback was visually provided on the experiment program. Participants could freely revise their criteria notes. These training and practice sessions were repeated once again to ascertain and deepen their identification criteria. In the main session, participants’ task was the same as that of the practice session, but there was no correct answer feedback. Participants could feel the stimulus as many times as they wanted.

Table 1 shows the numbers of repetition made for each stimulus. For each experimental condition, the stimuli were randomly distributed within four sub-sessions. A break of approximately 1 min was provided between sub-sessions. The experiment took 30 to 120 min, depending on the condition.

Data Analysis. For each experimental result, we computed the confusion matrix from the results of each participant and obtained its IT using the standard formulae in [13]. These individual \(IT_{est}\)s were used to compute the conditions’ mean \(IT_{est}\) and additional statistical tests.

2.2 Results and Discussion

The confusion matrices obtained in the experiment are shown in Fig. 3 for the vibrotactile stimuli and Fig. 4 for the thermal stimuli. The \(IT_{est}\)s are presented in Fig. 5 along with the maximum possible values of IT.

Fig. 3.
figure 3

Confusion matrices obtained with the vibrotactile stimuli.

Full size image
Fig. 4.
figure 4

Confusion matrices obtained with the thermal stimuli.

Full size image

Vibrotactile Stimuli. The \(IT_{est}\)s were 1.51 bits (SD = 0.07 bits), 1.48 bits (SD = 0.40 bits), and 1.56 bits (SD = 0.23 bits) for \(N=3\), 5, and 7. We conducted one-way between-subjects ANOVA to analyze the effects of the number of stimuli on \(IT_{est}\). The number of stimuli did not significantly affect \(IT_{est}\) (\(F_{2,18}=0.157,p=0.856\)). Thus, we can conclude that the \(IT_{est}\)s were saturated to 1.52 bits (the mean of the three \(IT_{est}\)s; 2.9 stimuli).

The vibrotactile stimuli had only one design variable of frequency. The \(IT_{est}\)s are comparable to those reported in the literature in similar settings [13].

Thermal Stimuli. The \(IT_{est}\)s were 1.72 bits (SD = 0.36 bits), 1.69 bits (SD = 0.26 bits), and 1.75 bits (SD = 0.36 bits). According to one-way between-subjects ANOVA, there was no significant difference in \(IT_{est}\) between the number of stimuli (\(F_{2,18}=0.068,p=0.935\)). Hence, the \(IT_{est}\)s were saturated to 1.72 bits (about 3.3 stimuli) regardless of the number of stimuli.

For further analysis, we calculated the percent-correct (PC) scores for different temperature change directions (cooling, no change, and warming). The results were 90.8, 88.5, and 84.3% for \(N=5\), 7, and 9, respectively. Thus, the participants could identify the direction of temperature change relatively well. Furthermore, the participants identified the cool stimuli with higher accuracies than the warm stimuli (see Fig. 4) because cold-sensitive thermoreceptors are distributed more densely than warm receptors in the human skin [5].

Fig. 5.
figure 5

Estimated and maximum IT values with standard errors.

Full size image

3 Exp. 2: Multimodal Stimuli

The aim of this experiment was to estimate the IT of multimodal stimuli combining vibrotactile and thermal stimuli, and then test the additivity law for IT to assess the level of perceptual independence between the two haptic channels.

3.1 Methods

Experimental Conditions. The experimental conditions were designed by combining the vibrotactile and thermal stimuli selected from Exp. 1 (Fig. 2). For the vibrotactile stimuli, \(IT_{est}\)s were saturated to 1.52 bits (2.9 stimuli). Thus, we chose three stimuli (slightly larger than the vibrotactile channel capacity) with frequencies 40, 126, 400  Hz. For the thermal stimuli, \(IT_{est}\)s were saturated to 1.72 bits (3.3 stimuli). We selected four thermal stimuli, also slightly larger than the thermal channel capacity, as −4, −2, 0, and +4 ℃/s (very cold, cold, no change, and warm). We included two cold stimuli because the cold stimuli were identified better than the warm stimuli in Exp. 1. Consequently, we used 12 experimental stimuli (3 vibrotactile and 4 thermal stimuli). The two types of stimuli were presented simultaneously with the same durations (1.5 s).

Participants and Procedure. Seven participants (3 males and 4 females; age \(M=\) 22 years and \(SD=\) 0.58) who did not participate in Exp. 1 took part in this experiment. The participants’ task and experimental procedure were the same as those of Exp. 1. The number of repetitions was 12 for each multimodal stimulus. The experiment took up to 120 min.

Fig. 6.
figure 6

Confusion matrices for the multimodal stimuli.

Full size image

3.2 Results and Discussion

The confusion matrix of the multimodal stimuli is shown in Fig. 6 (left). The \(IT_{est}\) was 3.06 bits (SD = 0.24 bits). This value is slightly less than the sum (3.24 bits) of the \(IT_{est}\)s of the vibrotactile and thermal stimuli obtained in Exp. 1.

For more detailed analysis, we show two confusion matrices for each modality Fig. 6 (right). Their \(IT_{est}\)s were 1.37 and 1.62 bits for the vibrotactile and thermal stimuli. These numbers are reduced to 90.0% and 94.2%, respectively, compared to the \(IT_{est}\) of the individual modalities.

Singhal et al. [10] found that the perception of thermal stimuli was affected by concurrent vibrotactile cues in terms of IT. Green [4] also reported a multi-sensory effect between the tactile and thermal stimuli: a cold stimulus impairs the sensitivity of high-frequency vibration and a warm stimulus lowers the vibrotactile sensitivity in a wide frequency range. Shim et al. [9] showed that the simultaneous presentation of wind and vibration stimuli remarkably reduce the accuracy and the IT of each unimodal stimuli compared to the sequentially combined ones. Despite such potential crossmodal interferences, the \(IT_{est}\) of the multimodal stimuli achieved over 90% of the information transmission performance of the individual stimuli. Therefore, the two modalities may not be completely perceptually independent, but combining them increases the identification ability of tactile stimuli to a large extent.

4 Conclusions

In this paper, we evaluated the IT of vibrotactile and thermal stimuli in two cases. The first case was when the two stimuli were provided separately. The \(IT_{est}\)s were saturated to 1.52 bits (perfect recognition of 2.9 stimuli) and 1.72 bits (perfect recognition of 3.3 stimuli) for the vibrotactile and thermal stimuli. The second case was when the two types of stimuli were provided simultaneously to the same body location. The \(IT_{est}\) was 3.06 bits (perfect recognition of 8.3 stimuli), which is approximately 90% of the sum of the individual modality \(IT_{est}\)s (3.24 bits). This result indicates that the degree of crossmodal interference between vibrotactile and thermal senses is approximately 10% in terms of IT, suggesting a high potential of combined vibrotactile and thermal stimuli for effective information delivery.

This study was preliminary in nature. Its positive results encourage us to explore this research space with improved depth and breadth. Our results can also be applied to contexts demanding high information transmission like gaming controllers and in-car infotainment systems. Our future work may consider introducing more tactile modalities, e.g., impact, and comparing the IT of their combination.