Keywords

1 Introduction

There are two conventional methods to experience music: listening physically (live) and listening through digital media. The method to listen to music physically is to go to a live venue or concert hall and listen to live performances by bands and orchestras. The method of listening through digital media is to listen to digitized sound through a medium such as a CD or a television. Typically, there are more opportunities to listen to music through digital media.

Live performance not only transmit sounds to the audience, but vibrations and sound pressure are also simultaneously transmitted. Conversely, when listening audio through digital media, it is difficult to feel vibrations and sound pressure on the body through general speakers or headphones. Therefore, the quality of the music experience greatly differs between these two methods. Furthermore, due to the difference in the quality of the music experience, it is hard to convey the music’s excellence through a digital medium. This makes it difficult to utilize digital method in entertainment such as music promotion and performance. Therefore, we thought that we could improve the quality of the music experience and use it for entertainment while listening to music through digital media.

It is believed that vibration that is felt by the body along with sound greatly affects the quality of the music experience. Since sound is air vibrations, especially large sounds and low frequency sounds can not only be heard by the ear but resonate through the body as vibrations. At live venues and concert halls, there are loud sounds and low frequency sounds, so the body feels vibrations strongly. However, general speakers and headphones cannot transmit strong vibrations. Furthermore, to improve entertainment, the listener need to feel as if he or she is enveloped in music, as if he or she was singing. This cannot be experienced by simply listening to music.

We attached speakers to the jacket, thereby creating a LIVEJACKET that presented a vibratory tactile sensation throughout the body, while one could hear the music. It is possible to play high band sounds and feel vibration by using multiple speakers including piezo speakers, full range speakers, and subwoofers. The listener can become more immersed into the music by mixing sounds exclusively for each body part.

This study’s contributions are as follows.

  • We experimented on methods to listen to music and found that emotions do not change significantly depending on how one listens.

  • We found that using the proposed method, LIVEJACKET, improves the music experience compared to other music listening methods.

  • The method of listening to music with headphones while vibrating the body also contributed to improving the music experience.

2 Related Work

Vibration tactile technology has already been used for the purpose of improving the quality of music experience. In addition, vibrotactile technology is also used in the field of entertainment such as games and movies. However, there are few studies on tactile presentation technology aimed at both improving music experience and utilizing it in the field of entertainment.

2.1 Improve Music Experience

There has been much research on the relationship between vibration tactile and music experiences [3]. Most of music playing devices that reproduce vibration are chair type devices. A chair-type device presents vibration by attaching transducers to a chair. The Emoti-Chair [7] converted acoustic signals directly into vibration by attaching eight voice coils to a chair. Merchel and Altinsoy [12] vibrated the entire chair and examined the change in the music experience. By individually controlling the sound and vibration to be played back, they clarified that vibration affects the music experience. Nanayakkara et al. [13] developed a chair-type vibratory sensation device for the hearing impaired. Experiments with hearing impaired individuals indicated that the music experience improved. Karam et al. [6] developed a jacket type vibration presentation device. Although no change in emotion was seen, it was observed that changing the vibration according to the music frequency improved the music experience better than adding a certain vibration.

Rovan [14] developed a haptic presentation system as an interface for playing virtual instruments. When playing a virtual musical instrument, vibrations are feedback, so that it feels as though the user is manipulating real instruments. In the research to improve the music experience using vibration tactile, music was played from the external speaker as before, and only vibration was presented from the haptic presentation device. To further improve the music experience, we played not only the vibration but also the music from the tactile presentation device. By playing music from the device as well, it sounds as if wearer is generating sound himself. Wearner feel as if he or she are being enveloped in sound, which leads to an improvement in the music experience.

2.2 Utilization in the Entertainment Field

Vibration tactile presentation devices are used not only in the music field but also in the entertainment field. As observed from the widespread use of 4DXFootnote 1, a chair-type vibration presentation system, vibration presentation is an important technique in movie viewing. Lemmens et al. [9] developed a jacket-type upper body vibration tactile presentation device for movie viewing. Dijk [4] has developed a blanket type vibrotactile presentation device for movie viewing with 176 vibrating motors. Mood Glove [11] is a globe type vibration tactile presentation device for viewing movies. It was determined that vibrotactile stimuli at low intensity and low frequency induced a sense of calmness in users, whereas vibrotactile stimuli at low intensity but higher frequency increased excitement. Abdur Rahman et al. [1] developed a wearable device that presents vibrations to correspond with movies published on YouTube.

Vibration tactile sense is also an important technology in gaming. Lindeman et al. [10] developed a vibrotactile presentation device for VR games. Synesthesia Suit [8] has developed a suits-type vibrotactile presentation device specializing in Rez Infinite games. It was used to promote of Rez Infinite. Surround Haptics [5] is a device that creates a sensation as if the vibrating part is moving between the transducers by arranging a plurality of transducers in an array in a chair; this is used to represent movement in the game. Emojacket [2], a jacket type haptic sense presentation device, presented not only vibrotactile presentation but also haptic sense by using air and temperature.

In the entertainment field there are few cases of using vibration tactile for music listening. We consider that this research will increase the entertainment aspect of the music experience.

Fig. 1.
figure 1

Layout drawing of the speakers to be attached to the jacket.

Full size image
Fig. 2.
figure 2

Block diagram of speakers and amplifiers used for LIVEJACKET.

Full size image

3 Implementation

LIVEJACKET consists of a jacket and 22 multiple types of speakers. Since the frequency band that can be played back is different depending on the speaker type, it is possible to reproduce broadband sound and vibration by using multiple speakers. We used 16 piezo speakers “pzBASS B26C” (PE1–16), four full range speakers “NSW 1-205-8A”Footnote 2 (SP1–4), and two subwoofers “SUBPAC M2”Footnote 3 (SB1 and SB2). We attached six piezo speakers in front of the jacket, three piezo speakers and two full range speakers in the right arm, three piezo speakers and two full range speakers in the left arm, and four piezo speakers and two subwoofers on the back (Fig. 1). Each speaker is connected to the audio interface MOTU 16AFootnote 4 that can output 16 channels via an amplifier. The piezo speakers are connected to a compact amplifier, full range speakers are connected to two 5 W amplifiers FOSTEX AP 05, and the subwoofers are connected to the attached dedicated amplifier (Fig. 3). The audio interface is connected to the laptop by a USB cable. Some speakers (PE1 and SB1; PE2 and SB2; PE15, SP1, and SP2; and PE15, SP1, and SP2) are connected to the same channel (Fig. 2).

The vibration and sound presented by LIVEJACKET needs to be dedicated. We split music for each part such as an instrument or a vocal and assigned the divided sound part to each channel of the audio interface using Logic Pro X. Since it is possible to play different sounds from each speaker, various sound designs are possible. It is possible to change the gain of each channel or to send the same sound from a plurality of speakers.

4 Evaluation

We experimented and evaluated how the quality of the music experience changed between five music listening methods.

4.1 Participants

Twelve participants (five females, seven males, eight of whom were members of our laboratory) between the age of 19 and 38 (M = 23.08, SD = 4.82) participated in the experiment. None of the participants were deaf. The average height of the participants was 168.5 cm (SD = 8.54).

Fig. 3.
figure 3

Experiment setup. Subjects wore the LIVEJACKET and listened to music while standing.

Full size image

4.2 Experimental Design

We prepared five music listening methods and two experimental music methods and compared the quality of the music experience. A questionnaire survey was conducted for each music listening method. Five kinds of music listening methods were prepared as follows. In all listening methods, subjects wore the LIVEJACKET and listened to music while standing (Fig. 3).

  • Speaker method: A method of listening to music using an external speaker. Subjects listened to music while standing 1.5 m in front of the stereo’s external speaker. We played the original, unprocessed sound source on the external speaker.

  • Headphone method: A method of listening to music using headphones. Subjects wore headphones. We played the original, unprocessed sound source on the headphones.

  • Speaker with vibration method: A method of listening to music using LIVEJACKET, which was set to present only vibration, and external speakers. We played the original, unprocessed sound source, on the external speakers. We mixed 100 Hz or 200 Hz low pass filtered sound source to each LIVEJACKET speaker and played. We played the sound source for LIVEJACKET and the external speaker simultaneously.

  • Headphone with vibration method: A method of listening to music using LIVEJACKET, which was set to present only vibration, and headphones. Subjects wore headphones. We played the original, unprocessed sound source on the headphones. We mixed 100 Hz or 200 Hz low pass filtered sound source to each LIVEJACKET speaker and played. We played the sound source for LIVEJACKET and the headphones simultaneously.

  • LIVEJACKET method: A method of listening to music wearing LIVEJACKET. We mixed each part of the original sound source for each speaker of LIVEJACKET and played.

Two kinds of songs were used in this experiment. To evaluate LIVEJACKET from the entertainment perspective, we selected rock and orchestra, as these music genres are often played at live venues. The song details and the mixing settings are listed below.

  • M1 - We are (ONE OK ROCK): For rock music, we selected “We are”Footnote 5 from ONE OK ROCK, a Japanese band, as rock music. We trimmed M1 from the original to 1 min 18 s. M1 is divided into 12 parts: bass, drums, percussion, rhythm, lead guitar, rhythm guitar, acoustic guitar, synth bass, synthesizer, string, chorus, and lead vocal. Bass and synth bass were output from PE1, PE2, SB1, and SB2. Vocals were output from SP1, SP2, SP3, SP4, PE15, and PE16, which are about the tip of the arm. By being able to hear vocals from the arms, it feels as though wearer are singing. The other parts were mixed so that the sound was reproduced from the whole body. In the experiment, subjects held their arm in front of his or her face like he or she had a microphone so that vocals reproduced from their arms could be heard from the face.

  • M2 - Symphony No. 5, Op. 67 (Ludwig van Beethoven): We selected Beethoven’s Symphony No. 5, Op. 67 for orchestra music. We trimmed M2 from the original song to 58 s. M2 is divided into 12 parts: trombone, trumpet, viola, violin, bassoon, clarinet, oboe, flute, cello, contrabass, horn, and timpani. Each part was associated with each speaker.

4.3 Procedure

Each participant was briefly informed of the study’s purpose and advised that they could abort the study and take a break at any time. Further, they were provided with a consent form to sign and a demographics questionnaire to complete.

Subjects listened to first to M1 then to M2 through the five types of listening methods. The order of five music listening methods was randomly presented to each subject. Both music selections were played once for each listening method.

We used Rousse’s circumplex model of emotion [15], a Multidimensional Scaling (MDS) model with two-dimensional interpreted parameters, arousal (weak to strong) and valence (positive to negative), to examine the subjects’ emotions after listening to music. Subjects responded to the MDS after listening to music with each listening method. They were asked the following questions, which were based on 7 scale likert scale. At the conclusion, we provided a free description field.

  1. Q1.

    Did the music feel comfortable? (comfortable - not affected - uncomfortable)

  2. Q2.

    Did the vibration feel comfortable? (comfortable - not affected - uncomfortable)

  3. Q3.

    How did you feel the music volume? (big - small)

  4. Q4.

    Did you feel the music to whole body? (overall - moderate - locally)

  5. Q5.

    Did you feel as if you were in a live concert venue? (I felt - I did not feel)

  6. Q6.

    Did you feel wrapped in sound? (wrapped - not wrapped)

  7. Q7.

    Did you enjoy it? (fun - not fun)

  8. Q8.

    Did you feel as if you became a singer/conductor? (I felt - I did not feel)

  9. Q9.

    Did you feel like you were in an audience? (I felt - I did not feel)

  10. Q10.

    Did you feel the sound pressure? (I felt - I did not feel)

  11. Q11.

    Were your emotions shaken? (shaken - not shaken)

  12. Q12.

    Did you want to move your body? (I wanted to move it - I did not)

  13. Q13.

    Did you want to sing along? Would you like to play in instrument? (I did - I did not)

  14. Q14.

    Did you want to go to a live venue? (I wanted - I did not)

  15. Q15.

    Would you recommend this to someone? (I would like to recommend - I did not want to).

4.4 Result

We conducted a statistical test on the questionnaire result using mauchly’s sphericity test and sidak for the multiple comparison. We used the SPSS Statistics version 24.

MDS results are shown in the Fig. 4. Significant differences were observed in arousal in M1 (F(4, 44) = 3.540, p < 0.05). As a result of multiple comparison, the LIVEJACKET method showed higher awareness than the speaker method (p < 0.05). Conversely, the valence in M1 did not show any significant difference (F(4, 44) = 1.286, n.s.). In M2, there was no significant difference between arousal and valence (arousal: F(4, 44) = 3.508, n.s.; valence: F(4, 44) = 1.999, n.s.).

We conducted a statistical test on 15 questions as well as MDS. Only the results on the question items with significant differences are shown in the Fig. 4. Significant differences were found in Q4, Q5, Q6, Q7, Q8, Q10, Q11, Q13, Q14, and Q15 in M1 (Table 1). Significant differences were found in Q4 and Q10 in M2 (Table 2).

Table 1. Result of statistical test on the questionnaire result using multiple comparison in M1.
Full size table
Table 2. Result of statistical test on the questionnaire result using multiple comparison in M2.
Full size table

The results of the multiple comparison are as shown in the Fig. 4. Characteristic results indicated the following. From the result for Q4 in M1 and M2, we proved that the LIVEJACKET method can create the sensation of music being felt throughout the body. However, the headphone with vibration method also proved to be effective to some extent. According to the result of Q8 in M1, we found that subject could feel as if he or she became a singer by using the LIVEJACKET method, speaker with vibration method and headphone with vibration method more than the speaker method. In Q4, Q5, Q6, Q7, Q8, and Q11 of M1 and Q4 and Q10 of M2, not only the LIVEJACKET method but also the headphone with vibration method showed significant differences.

P1 answered that jacket was heavy and tiring. P3, P4, P5, P10, and P11 answered that it would be better to strengthen the front vibration, not the back. P8 answered that he felt music playing in the back. P2 and P4 answered that the sound quality of LIVEJACKET was bad.

Fig. 4.
figure 4

The result of MDS model of emotion and questions about emotion. Only the result on the questions with significant difference are shown.

Full size image

5 Discussion

We conducted a questionnaire survey on MDS and music experience in subject experiments. In the MDS evaluation, it was suggested that there was no significant difference in emotions with each music listening method. This indicates that the differences in listening methods do not affect emotions. It is thought that the tune of the song itself subject’s affects. The rock music in M1 had higher arousal and valence than M2, which is an orchestral piece.

In the questionnaire survey on the music experience, there were significant differences in many questions. From M1, there were significant differences with 10 questions, however with M2, there were significant differences with only two questions. This is thought to be due to the difference in songs. As P2 and P4 answered, LIVEJACKET has poor sound quality compared to external speakers and headphones. As for an orchestra, which has many kinds of musical instruments and high resolution, sound quality changes greatly depending on the device to be reproduced. Therefore, when playing the orchestra with LIVEJACKET, it is thought that the sound quality decreased more than with external speakers and headphones, which affected the music experience. However, since M1 is a kind of rock music with fewer musical instruments, the influence of difference in sound quality is considered to be small. In the seven questions, significant differences also appeared in the headphone with vibration method in addition to the LIVEJACKET method. This is also considered to be related to sound quality. Compared to the LIVEJACKET speakers, the headphone sound quality is better. Furthermore, the position of the LIVEJACKET loudspeaker is closer to the subject than the external speaker, but similarly the sound source from the headphone is closer to the subject. According to these points, it is considered that the music experience also improved in the headphone with vibration method in which the sound quality was good and the sound source was close to the subject.

A questionnaire survey about the music experience indicated that LIVEJACKET can provide a new music experience. By wearing the LIVEJACKET, subjects were able to experience music as it would be originally felt in a live venue, such as feeling music throughout the whole body and feeling sound pressure. Furthermore, subjects felt like they became singers. Subjects felt that they wanted to go to a live venue. These experiences were hard to feel when listening to music through conventional digital mediums. From these results, we can consider about how to use LIVE JACKET. For example, we can use LIVEJACKET for entertainment so that wearer can feel the experience of being a singer. The jacket can also be used to promote live participation.

According to the subjects, it would be better to strengthen the vibration in the front side, not just the back side. Because the current system has a subwoofer on the back, the vibration felt in the rear side is strong. However, subjects mostly go to the live venue as an audience, so speakers and musical instruments that generate sound exist in front of the subjects. For that reason, we feel intense vibration on the front body in a live venue. LIVEJACKET vibrates strongly at the back, so they feel the difference between being in a live venue and wearing the LIVEJACKET. This is why they suggested that the front vibration be stronger. To bridge this experience gap, it is necessary to install a speaker that generates intense vibration such as a subwoofer in the front as well as on the back.

6 Conclusion

We developed LIVEJACKET, a jacket capable of vibrotactile presentation and playing music. In this paper, we conducted experiments with five music listening methods to investigate the quality of the music experience. We found that LIVEJACKET creates a higher quality music experience than other listening methods. LIVEJACKET is suitable for music which has fewer instruments. In the discussion, we examined how to utilize LIVEJACKET in promotion and entertainment. To convey more of the music’s excellence, we need to raise the quality of the music experience. We believe that LIVE JACKET can fulfill this objective.