Keywords

1 Introduction

Current 3D digital film gives us a more realistic sensation. However there is still some problem that keeps us away from immersing the horror contents. In order to find an effective way to amplify horror emotion to viewers, we propose cross modal display system to enhance horror emotion. As a first step, we developed a pseudo heartbeat feedback system to give vibrotactile feedback. In concrete, we made a locker-type 3d movie environment while generating heartbeat-like vibration on the sole of the foot. We conducted the experiment to view the horror movie with the system. In this experiment, we propose two methods to generate pseudo heart beat vibration. One is to raise heartbeat vibration by referring a user’s heart rate real time. Another one is to raise heartbeat vibration in a stepwise manner up to predetermined heart rate value. We evaluated whether our proposed system effects to change viewers’ real heart rate and analyzed which type of method is effective to change its real heart rate by comparing ration of synchronization.

2 Related Works

2.1 Cross Modal Stimuli and Emotion

As for utilizing crossmodal effect for emotion amplification, Furukawa et al. proposed the system to control piloerection on the forearm artificially to generate the feeling of surprise while playing audio or watching movie [1, 2]. Coen et al. conducted an experiment to evaluate a link between negative emotional state and abnormal visceral sensation from the point of brain processing [3]. As experimental settings, they compared brain process of a subject listening to emotionally negative music with artificially giving stimuli into its distal esophagus and concluded that outer stimuli into esophagus amplifies negative emotion.

2.2 Biofeedback and Emotion

Ohkura et al. clarified the relationship between emotional feeling such as surprising or exciting feeling and change of biological signals such as ECG (electrocardiogram), SPA (skin potential activity) to derive the Kansei model to evaluate emotional state objectively. [4] One of their researches described that the fearful feeling was influenced by change of visual stimuli by evaluating the raise of heart rate [5].

The interactive art work called “Empathetic heartbeat” generates empathetic feeling by watching movies while hearing its own heartbeat being amplified through the stethoscope on its chest [6]. To overlay heartbeat coming from its own body is effective to imagine other people’s emotion in the movie and is immersed in a story.

These previous research and work suggest that emotion and biosignal are well related and thus biofeedback will be effective to change its emotional state from outer stimuli. Our previous research also concluded that watching horror movie in a closed space raised heart rate significantly rather than watching it in an open space [7] (Fig. 1).

Fig. 1.
figure 1

Horror movie capture image

Full size image

Based on these findings, we developed a cross modal system to experience horror movie with vibrotactile feedback of pseudo heartbeat simulated by its own heart rate in a closed box. We evaluated if there is emotional change by watching the horror movie with the system. In this paper, we evaluated if vibrotactile feedback of pseudo heartbeat draws real heart beat faster.

2.3 Synchronization Phenomena

Synchronization phenomena is an interacting elements existing in the world. For example if a large number of fireflies gather around a single tree, they will eventually start to flash at the same time creating a bright flashing light. Similarly we experience in a concert halls a large number of hand claps sound separately but synchronize eventually. This synchronization phenomena is mathematically defined with so called the Kuramoto model [8]. This model is defined as the Eq. 1.

$$ \frac{d\phi i}{dt} = \omega_{i} + \frac{K}{N}\sum\limits_{j = 1}^{N} {\sin \left( {\phi_{i} - \phi_{j} } \right)} $$
(1)

where a population N coupled phase oscillators in i is φ i and its variable velocity is φ j having natural frequencies ω i distributed with a given probability density K of dt. We adapt this model to evaluate the effectiveness pseudo heartbeat frequency to change real heartbeat frequency. We calculate the rate of synchronization between the pseudo heartbeat frequency and its real heartbeat frequency in order to evaluate the effectiveness of these two biological rhythm.

3 Horror Emotion Amplification System

We developed a prototype horror emotion amplification system. The system configuration is shown as Fig. 2. The system simulates a locker box with peeping holes on the front panel with paralyzed film. This is a same design with our previous research in [7]. By peeping a hole, a person watches 3D movie without wearing a paralyzed glasses. With our new prototype system, we added a vibrotactile display on the bottom in order to give feedback of real/pseudo heartbeat. As Fig. 3 shows, we implemented a subwoofer (Buttkicker BKA-113-C) under 50 cm(w) × 50 cm(d) × 20 cm(h) wood plate. By standing on it, a user feels generated heartbeat through bottom of feet. We expect echoing its own heartbeat produces as if he were locked in a closed area. We developed monitoring application which records its interval time of heart beat, edits heart beat timing and outputs vibration of edited heart beat through the vibrotactile display while capturing his heart rate with a photodiode heart rate sensor real time. Figure 4 shows a captured display of the monitoring application.

Fig. 2.
figure 2

System diagram

Full size image
Fig. 3.
figure 3

Vibrotactile feedback system bottom (left), front (right)

Full size image
Fig. 4.
figure 4

Monitoring display of heart rate

Full size image

Figure 5 shows an overall system we implemented. 2 m × 6 m silver screen and vinyl pipes’ screen frame are constructed with four persons at site. Two rear projectors are set on the back of the screen for 3d projection.

Fig. 5.
figure 5

Experiment Setting

Full size image

Figure 6 shows a locker-like box. The size of the box is 50 cm × 50 cm × 200 cm. One person is able to stand inside a box while peeping holes in front for watching a 3d movie played on the silver screen outside. A person’s heart rate is captured on the middle finger of right hand by a photodiode sensor (Fig. 6 top right). A frequency pattern of interval of heart beat is generated according to the captured heart rate and sent to vibrotactile display (Fig. 6 bottom right).

Fig. 6.
figure 6

Locker-like box and its attachments (photodiode sensor (top right), subwoofer speaker (bottom right))

Full size image

4 Pseudo Biofeedback Experiment

The purpose of the experiment is to evaluate if the system amplifies horror experience and to analyze two methods of generating pseudo heartbeat is effective to change the frequency of real heart rate and evaluate which is more effective. In this paper we used rate of synchronization between real heart rate and pseudo heart rate to compare the effectiveness between two proposed conditions.

Horror movie we used for the experiment was the same one in the previous research [7]. The scene of the movie was categorized into 14 scenes. We followed the categories and displayed real and pseudo heart rate calculated by two conditions.

The conditions to generate pseudo heart rate is described as follows.

The first one is to raise pseudo heart rate interactively adapting a subject’s heart rate real time. The second one is to raise pseudo heart rate in a stepwise manner up to the predetermined heart rate. As a control condition, no vibriotactile stimulus was given to one third of the total subjects. Table 1 shows fluctuation conditions and Table 2 shows conditions given in each categorized scene.

Table 1. Conditions of pseudo heartbeat feedback
Full size table
Table 2. Horror movie scene description
Full size table

As the previous finding in [7], the maximum raised heart rate of subject’s heart rate is 20 bpm larger than mean heart rate while it watches about 120 s’ highlight of the same horror movie in a closed locker in the upright position. We concluded that this number is biologically boundary of fluctuation of heart rate for 120 s as a human being in the upright position. So we fixed the maximum increase of pseudo heart rate is 20 bpm larger than mean heart rate of each subject. So if a subject’s mean heart rate is 70 bpm, we did not raise pseudo heart rate more than 90 bpm.

Pseudo heart rate (fBPM) is calculated by Eq. 2. Elapsed time after giving vibrotactile stimulus is defined as \( \varDelta t \). Average heart rate at rest is defined as aBPM. The subtraction of average heart rate for 10 s of each subject and generated pseudo heart rate is defined as dif. Also we simulate the raise of heart rate is similar to quadratic function not linearly which means that heart rate raises gradually. So by the quadratic function 20 = a120(sec) 2, we led coefficient a as 720. Dif is ignored when to calculate the pseudo heart rate of condition 2.

$$ fBPM = 60000\times( {\frac{720}{{\varDelta t\left( {\sec } \right)^{2} + 720\times\alpha BPM}}}) - dif $$
(2)

30 university students (male 17 female 13) participated the experiment and each 10 subjects did the experiment with one of the conditions. The experiment was conducted one by one in a darkroom. A subject entered a box and stood still while looking at point of gaze on the screen for four minutes. We recorded the heart rate of this four minutes as rest time and the mean heart rate was calculated to determine maximum value of raising heart rate. After the four-minute rest time, the horror movie was played as well as vibrotactile stimulus was displayed. The first 101 s from the scene 2 to 4, real heart rate of each subject was used to generate vibrotactile stimuls in order to get accustomed to the stimulus on the foot. From scene 5 to 10, the pseudo heart rate was calculated to generate vibrotactile stimulus. After the movie, subject stood still for two minutes to record heart rate as after rest.

5 Result

Figure 7 shows ratio indicator of mean heart rate of each scene and mean heart rate of rest time. From scene 5 (zombie appeared), pseudo heartbeat was displayed by the vibrotactile display. And the highlight of the movie was scene 10 (zombie close-up). We did not find any change of heart rate in all conditions when the movie started. However in condition 1, from scene 6 (another friend is killed) heart rate raised gradually and during the hightlight scene the heart rate raised maximum value during the movie was played. Also after the movie the heart rate still kept increasing. The maximum value of all scenes was scene 12 (rest(31–60 s)). In condition 2, scene 7 (zombie approched) was maximum heart rate during the movie not the highlight scene. And the maximum value of all scenes was scene 11 (rest (− 30 s)). In condition 3, the highlight scene was the maximum value of the heart rate of all scenes but right after the highlight, heart rate declined.

Fig. 7.
figure 7

Mean heart rate ratio

Full size image

We also analyzed rate of synchronization of condition 1 and 2 in order to evaluate which method effects to change real heart rate. Kuramoto order parameter was calculated by Eq. 3. This parameter affects the strength of binding of real and pseudo heart rate. The phase of heart rate (HFrequency = \( \theta \)) was calculated by Eq. 4. Inter-beat interval is defined as IBI as well as fake inter-beat interval (fIBI) and based on last 128 (f)IBIs, mean \( \overline{(f)IBI} \) was calculated.

$$ SyncRatio = \frac{1}{2}\left| {\sum {e\hat{i}\theta } } \right| $$
(3)
$$ HFrequency = \frac{{\mathop {2\pi \times IBI}\nolimits_{i} }}{{\overline{IBI} }} $$
(4)

Figure 8 shows mean rate of synchronization of each scene. If the rate is close to 1.0, rate of synchronization between real and pseudo heart rate is linked, which indicates that vibrotactile stimulus effects real heart rate.

Fig. 8.
figure 8

Rate of synchronization of each scene of condition 1 and 2

Full size image

Rate of synchronization in condition 1 is larger than that in condition 2. We applied t-test and there is significant difference in scene 8 (p < 0.05), scene 10 (p < 0.01) and scene 11 (p < 0.01). Even though there is no significant difference, scene 6, 7 and 9 also tend to be different (p < 0.1). This indicates to raise pseudo heart rate interactively adapting to real heart rate and feedback vibrotactile stimulus to a subject effects to change real heart rate more than another method.

6 Discussion and Future Works

As a first step of the research, we developed a pseudo heartbeat feedback system to give vibrotactile feedback. In concrete, we made a locker-type 3d movie environment while generating heart beat-like vibration on the sole of the foot. We conducted the experiment with the system. In this experiment, we gave two types of pseudo heart beat vibration. One is to raise heart beat vibration by referring a user’s heart rate real time. Another one is to raise heartbeat vibration in a stepwise manner up to predetermined heart rate value.

We evaluated whether our proposed system effects to change viewers’ real heart rate and analyzed which type of method is effective.

In conclusion we found

  1. (1)

    By giving vibrotactile feedback, it is likely to give afterglow of horror experience.

  2. (2)

    By generating pseudo heart rate synchronizing user’s heart rate real time, it is more effective to change one’s real heart rate.

From these results, it is possible to control one’s horror emotion with the combination of audiovisual and vibrotactile feedback. In this experiment, in order to find an effective method to use vibrotactile feedback as crossmodal stimuli, we hypothesized the raise of heart rate is similar to quadratic function. But it is still not clear this method is adequate to simulate raise of heart rate. Now we are interested in adapt Kuramoto synchronization model to simulate heart rate by referring a user’s heart rate real time. In future work we plan to adapt Kuramoto synchronization model to generate pseudo heart rate and analyze its effect. Also we will add olfactory stimuli in addition to visual and vibrotactile stimulus to evaluate which combination of crossmodal stimulus is effective to amplify the horror emotion.