Keywords

1 Introduction

Mobile devices such as cellular phones and smartphones notify their users of incoming calls or messages by using a vibrating alert. Tactile stimuli such as vibrations were first considered as a medium of communication in the 1950s. For example, Geldard’s fundamental article [4] about the perceptual characteristics of vibration stimuli referred to vibratory intensity discrimination, temporal discrimination, and learning curves for communication using “vibratese language.” The letters or some prepositions of the “vibratese language” were made by using five thoracic buzzes representing vibration stimuli composed of two dimensions, i.e., intensity and duration. In this way, the researchers focused on improving the efficiency of the communication through tactile and haptic sensory modalities. More recently, users of the latest mobile devices are able to create their own vibration patterns for different notifications from the device by tapping on the touchscreen. Our previous study involved using a smartphone to test and evaluate the idea of creating and using vibration patterns on such mobile devices [7]. In these studies, the vibration patterns were of constant duration and intensity, similar to Morse code, and the respective vibration patterns presented different notification messages such as “The battery is running out”, “You have a call”; therefore, the participants had to learn and memorize the correspondence between the vibration pattern and the corresponding notification message in advance. The experimental results suggested that the number of vibration patterns participants were able to recognize could be fewer than five; however, we found that it was a cue to allow memorable vibration patterns to be created such that they are characterized by the pronunciation of the notification message. In this regard, we observed that most of the participants used to create their own vibration patterns based on their respective tempi and rhythms when they spoke the message. These results were experimentally confirmed by another study using a custom-built vibration mouse including a vibrating motor. However, the pronunciation factors that affect the recognition of the vibration pattern as a specific message were not clarified [6].

As to the Japanese language, we not only considered phonograms in which the pronunciation and the letter coincide but also ideographs that have their own meanings. The form of an ideogram suggests the meaning is the key to understanding the message conveyed by the letter. The ideograms and the meanings are cued to remember each other. From this point of view, ‘tactons’ or ‘tactile icons’, from the concept of the ideograph, are a sort of vibration stimuli created from audial rhythms. Tactons have been researched from both the physiological and sensory points of view [9, 10]. On the other hand, a message written in Japanese phonograms such as Kana represents the pronunciation of the message. Thus, according to our previous studies, comprehensible vibration patterns that represent the pronunciation of a message could be used to approximate the phonological vibration pattern. Therefore, we have been researching the requirements for creating comprehensible phonological vibration patterns.

Japanese phonetics is based on the concept that the phonetic characteristics include accent, rhythm, intonation, tempo, and so on [4]. Thus, from the perspective of phonetic characteristics, we tried to make and evaluate vibration patterns based on the tempo of pronunciation [6, 7].

This study proposes using vibration patterns with the phonetic characteristics of the accent of pronunciations to determine the requirements of designing comprehensible vibration patterns. Thus, two types of vibration patterns, namely tempo-based vibration patterns and accent-based vibration patterns were compared by experimentally obtaining the participant’s tactile comprehension using a vibration mouse. In this regard, the method according to which the tempo-based vibration patterns are designed was proposed in the previous study [6]. Thus, we designed the tempo-based vibration patterns based on the existing method. We produced the accent-based vibration patterns by creating vibration patterns that were similar to the pronunciations using our custom-built controller for the vibration mouse and then evaluated the patterns by assessing participants’ tactile comprehension experimentally.

2 Evaluation of Tempo-Based and Accent-Based Vibration Patterns

2.1 Method

Vibration Mouse and the Control System.

Although we evaluated the vibration patterns using not only the vibration mouse but also a smartphone in our previous studies, the characteristics of memorable vibration patterns for the participants were about the same for both of these tactile interfaces. However, the vibration mouse is able to generate a vibration wave with larger amplitude, which is controlled by changing the voltage applied to the vibrating motor in the vibration mouse. Therefore, we used the vibration mouse to produce a wider range of vibration patterns in this study.

The vibration mouse was made by attaching a vibrating motor (S.T.L.JAPAN CL-0614-10250-7) to a computer mouse (DELL MO56UOA) and is able to present vibration stimuli with an averaged vibrational wave velocity of 2.3 mm per second. The power voltage for activating the vibrating motor attached to the mouse was controlled using a high-precision analog I/O terminal (CONTEC AIO-160802AY-USB) and a personal computer (DELL XPS 8300) running Windows 8.1 Pro Japanese edition. The voltage applied to the vibration motor was controlled by the I/O terminal with our custom software. In addition, the vibration mouse simultaneously functioned as the computer mouse with two buttons and a scroll wheel.

Designing Two Types of Vibrational Notification Messages.

Two types of vibration pattern were evaluated. The tempo-based vibration pattern should be composed of vibrations of constant duration and intensity as shown in the lower portion of Fig. 1; however, the amplitude of the vibrational wave velocity measured at the surface of the vibration mouse was rough with a delay according to the digital oscilloscope (Tektronix DPO2024B) to which a vibrometer (MotherTool VB-8205SD) was attached as vibration sensor as shown in the upper portion of Fig. 1. Although the respective vibration durations and the gap between the durations were decided based on a young individual’s reading of the message, the duration of vibrations was chosen from 100, 200, and 400 ms based on a previous study [8].

Fig. 1.
figure 1

Example of a vibrational waveform on the surface of the vibration mouse (upper). The vibration is rough and the amplitude is varied in spite of applying power voltage constantly to the vibrating motor (lower).

Full size image

On the other hand, the accent-based vibration patterns were designed according to an accentual dictionary of the Japanese language [1] and other reference books on Japanese linguistics. Concretely, we estimated the perceptual duration of vibration at 180 ms per mora based on the average duration of mora in Japanese reading. In the same way, we decided that the gap between vibration durations for choked sound was 240 ms and the duration of a long vibration was 380 ms. Further, the voltage applied to the vibrating motor was classified into three levels according to the alteration in accent. Therefore, the vibration waveform on the surface of the vibration mouse in case of presenting an accent-based vibration pattern was as shown in the upper portion of Fig. 2.

Fig. 2.
figure 2

Example of voltage shifts for presentation of the accent-based vibration pattern (lower) and observed vibrational waveform on the surface of the vibration mouse (upper). This example shows the vibration pattern for presenting the same notification message that is shown in Fig. 1.

Full size image

Our experiment involved creating two sets of vibration patterns and each set included the same six notification messages in advance. The two sets of vibration patterns were designed based on the method of designing tempo-based and accent-based vibration patterns of each. The six notification messages were Japanese sentences representing the following: “The bath is ready,” “The battery has been recharged,” “You forgot to turn off the light,” “You’ve got mail,” “Clean your kitchen,” and “The bus is due to arrive.”

Although we researched vibratory patterns from the view of vibratory perception, in practice, the vibrating sound normally perceived in the context of using the vibration mouse is produced on a desk or a table. In other words, the vibration patterns would not only be perceived by the user’s vibratory perception but also by their auditory perception. Therefore, in this experiment, the participants tried to evaluate the vibration patterns without covering their ears.

Participants.

We selected 33 male students ranging from 21 to 24 years of age and 3 female students ranging from 21 to 23 years of age as participants. The users of the vibratory interfaces had to be able to perceive the vibration patterns; however, one male student could not perceive the vibration pattern correctly through the experiment. Therefore, the experimental data of 35 students were analyzed in this experiment.

Experimental Procedure.

The experiment was applied within-subjects design and the participants tried to correlate the notification messages with the tempo-based vibration patterns in the first trial, whereas the second trial was used for evaluating the accent-based vibration patterns. The procedure followed during each trial was divided into two steps. Firstly, participants were expected to learn the correspondence between the notification messages and the vibration patterns using the software we developed. The software featured a learning window and a trial window. The learning window included six buttons indicating the different notification messages and a “Next” button in the lowest part of the window as shown in Fig. 3. Using this window, the participant was able to produce the vibration patterns on the vibration mouse by using the mouse to select a button on the screen. Participants were able to learn the correspondence between the vibration patterns and the notification message repeatedly until they internalized the relations. After the learning procedure, the participants clicked the “Next” button to progress to the next step upon which the learning window on the screen was replaced with the trial window as shown in Fig. 4.

Fig. 3.
figure 3

Learning window for confirming the correspondence between the notification message and vibration pattern.

Full size image
Fig. 4.
figure 4

Trial window for testing six vibration patterns

Full size image

After relaxing a while, the six vibration patterns were randomly presented to the participants one by one. The participants tried to correlate the messages with the vibration patterns by using the vibration mouse to select the answer from six notification messages or by selecting “I could not recognize it” in the right column of the trial window as shown in Fig. 4. Participants’ answers were recorded by the software and their opinions about the ease of learning and ability to recognize the vibration patterns were further assessed in an interview after all the trials.

2.2 Results

Number of Correct Answers.

We measured the comprehensibility of the notification message communicated by the vibration pattern by counting the number of correct answers by the participants for the two different kinds of vibration patterns: tempo-based and accent-based. As a result, the number of correct answers in the case of using tempo-based vibration patterns averaged 3.63 (SD = 1.43), although the number of correct answers in the case of using accent-based vibration patterns averaged 3.86 (SD = 1.56). The result of a two-sided test of statistical significance using a level of significance of 0.05 indicated that the difference between those averages were not statistically significant (p = .55). Thus, we concluded that the comprehensibility of both types of vibration patterns was almost the same.

Participants’ Opinions.

The outcome of the interviews conducted with participants determined that 15 of the 35 participants believed that tempo-based vibration patterns were comprehensible. Meanwhile, 20 of the 35 participants preferred the accent-based vibration patterns because of comprehensibility. As far as the characteristics of the pronunciation of notification messages are concerned, 32 of the 35 participants said that the vibration pattern representing the notification message including choked and/or long sounds was easy to learn regardless of the types of vibration patterns.

As for the reason for their preference for vibration patterns of either one of the two types, most of the participants not only pointed out the easiness to learn the correspondence between the notification messages and the vibration patterns but also mentioned the ease of imagining the pronunciations from the vibration patterns after learning.

2.3 Discussion

Although the number of correct answers for the respective types of vibration patterns was almost the same, participants’ opinions suggest that the ease with which the pronunciation of the notification messages is imagined is an important factor in designing vibration patterns. In addition, the result of our previous study suggests that the pronunciation of the notification message depended on the person; therefore, the vibration patterns representing the pronunciation should be designed following the user’s pronunciation. However, the vibration patterns for the experiment in this study were not designed according to the pronunciation of respective participants; instead, they were designed based on the pronunciation of a young individual. Further, the easiness of imagining the pronunciations from the vibration patterns varies between individuals. That is, for some people, neither of these types of vibration pattern would provide a sufficient cue to imagine the pronunciations of notification messages. In this regard, the above results confirmed our previous knowledge, even though the previous study only included the vibratory perception of the vibration pattern.

This explains why the number of correct answers for both types of vibration patterns seem to be almost the same. Nevertheless, it is difficult to describe both of the types of vibration pattern as being comprehensible.

In addition, the number of correct answers was almost the same as in our previous studies [6, 7]; therefore, it seemed that the vibrating sound of the vibration mouse has an insignificant effect on the perception of vibration.

2.4 Summary of Evaluation

Although the above results did not enable us to clarify a comprehensible type of vibration pattern, we found that neither the tempo-based nor the accent-based vibration patterns have sufficient cues to represent the pronunciation of the notification message. This suggested the need to develop a method to design a more comprehensible vibration pattern and encouraged us to design a new vibration pattern including more cues. Our aim was therefore to imagine the pronunciation of notification messages using a vibration pattern that includes characteristics of both tempo-based and accent-based vibration patterns.

3 Designing Vibration Patterns Based on the User’s Voice

We designed more comprehensible vibration patterns by focusing on the voice of the user of the vibration mouse, because voice or pronunciation could include characteristics of both the tempo and accent of speech. However, it is necessary to develop a new method to reflect the characteristics of the pronunciation on the vibration pattern. In this regard, one approach to creating a vibration pattern would be to convert the voice into vibratory stimuli. Thus, we developed a converter that amplified the speech signal and applied voltage to the vibrating motor in the vibration mouse. The vibration pattern produced by the converter was named “voice-based vibration pattern” and was evaluated together with the other two types of vibration patterns from the viewpoint of comprehensibility.

3.1 Developing a Converter for the Voice-to-Vibration Pattern

The new system for generating vibratory stimuli by the vibration mouse consisted of a capacitor microphone (SONY ECM-PCV80U), a personal computer (DELL XPS8300), the converter we developed, and the vibration mouse. The circuit of the converter was a modification of the circuit of a sound level meter as shown in Fig. 5.

Fig. 5.
figure 5

Circuit diagram of the converter

Full size image

The converter was driven by both the audio signal from the headphone jack and 5 V supplied via the USB (universal serial bus) port of the personal computer. The audio signal was output by playing a recorded audio file using our custom software and the microphone. Thus, the user had to record his/her pronunciation of notification messages and create the audio files in advance before presenting the vibration patterns as described later. The audio file consisted of data generated by analog-to-digital conversion at a sampling rate of 44.1 kHz with a 16-bit quantization rate.

The voltage applied to the vibrating motor was supplied from the output jack of the converter. Figure 6 shows the change in the waveform inputted as an audio signal including the pronunciation of the notification message to the vibrational waveform on the surface of the vibration mouse. The waveforms in Fig. 6 were measured and recorded by the digital oscilloscope at both points A and B of the circuit; however, the vibratory waveform C was measured by attaching the vibration sensor of the vibrometer. Comparing the waveform of the audio signal and the amplitude of vibration on the vibration mouse indicated that the tempo and accent mostly seem to be the same, though the amplitude of vibration has a lag time compared with the waveform of the audio signal.

Fig. 6.
figure 6

Example of waveforms recorded at points A and B in the circuit of the converter and the amplitude of vibration on the surface of the vibration mouse (indicated as waveform C).

Full size image

3.2 Creating Three Types of Vibration Patterns

We evaluated the voice-based vibration patterns by using eight notification messages consisting of the six notification messages used in the former experiment and the additional messages “Clean your toilet” and “The TV program is starting.” These eight notification messages were converted into three types of vibration patterns, namely tempo-based, accent-based, and voice-based. However, the voice-based vibration patterns had to be produced by the respective participants in advance of their trials using our custom software.

On the other hand, the tempo-based and accent-based vibration patterns were created based on the pronunciation of the young individual who was used in the former experiment.

3.3 Experiment

The experiment was with-in-participants design and conducted for three conditions using tempo-based, accent-based, and voice-based vibration patterns for the same eight notification messages. The participants were required to evaluate the comprehensibility of the three types of vibration patterns. However, the vibration patterns were not only evaluated in terms of their comprehensibility but also from a learnability perspective in the context of daily use. In other words, we assumed vibration patterns requiring more understanding from the user as being less effective. In this regard, the data would not provide a clear indication as to the learnability if participants were given opportunities to understand the correspondence between the vibration patterns and notification messages as far as they considered it necessary. Thus, participants were intentionally allowed few opportunities to understand the correspondence between the vibration patterns and the notification messages before the trials in this experiment.

The software for learning and evaluating tempo-based and accent-based vibration patterns was used as well as the experiment described before. However, this time the window for software learning included eight buttons indicating the different notification messages. Each button was only available once and the software presented the vibration pattern on the vibration mouse only once. As for learning and evaluating voice-based vibration patterns, we employed additional custom-built software for use in the experiment. The software had three functions: recording the user’s speech, understanding the vibration patterns based on the recorded speech, and evaluating the vibration patterns one by one.

Procedure.

In the experiment, the participants repeatedly evaluated the eight tempo-based, accent-based, and voice-based vibration patterns; therefore, participants repeatedly attempted understanding and correlating the notification messages with the vibration patterns three times in a similar way as follows. In addition, participants were required to listen to white noise with a headphone because we had little idea of the tactile perception stimulated by the voice-based vibration patterns.

Firstly, participants evaluated the tempo-based vibration patterns following the same procedure used in the former experiment. More specifically, after the participants experienced all eight vibration patterns one by one, they tried to randomly correlate the eight notification messages with the vibration patterns presented and answered the message they recognized by choosing from the list of eight notification messages and clicking the answer on screen using the vibration mouse.

Secondly, the accent-based vibration patterns were learned and evaluated in the same way as the tempo-based vibration patterns.

Lastly, the subjects recorded the eight notification messages for presenting the corresponding voice-based vibration patterns using the microphone and the custom-built software as shown in Fig. 7. Participants were required to check the voice-based vibration patterns after recording their speech of the notification message and were able to try to record again until they were satisfied with the voice-based vibration pattern they produced. Further, they tried to tune the volume level using a keyboard, because the amplitude of the voice-based vibration pattern converted directly by the converter was changed according to the volume level of sound. About this point, the volume of the voice depended on the participant’s voice; therefore, quiet talkers were required to increase the volume level in order to present voice-based vibration patterns more clearly.

Fig. 7.
figure 7

Scene showing a participant recording his speech of eight notification messages one by one using the custom-built software.

Full size image

Before the trials, participants recorded their speech of the eight notification messages after which they had one opportunity to check or learn the respective voice-based vibrational message based on their speech converted by the converter. After relaxing a while, they tried to correlate the notification messages with the randomly presented voice-based vibration patterns and provided their answers on the screen. After completing all the trials, participants were interviewed to obtain their opinions about the effectiveness of the vibration patterns.

Participants.

The participants were 19 male students ranging from 21 to 23 years of age and 11 female students ranging from 19 to 22 years of age. Although all participants were physically unimpaired and were able to perceive the vibration stimuli from the vibration mouse, two participants admitted to finding it difficult to understand any of the notification messages of the vibration patterns. Further, an additional participant was unable to create voice-based vibration patterns at maximum speech volume because she was a silent speaker. Therefore, the experimental data of 27 participants were analyzed in this experiment.

Result and Discussion.

Based on the experimental data, the correct answers for each type of vibration pattern were averaged and compared as shown in Fig. 8. Further, we analyzed the variance and the difference among the three types of vibrations. The results showed that the difference was statistically significant (F(2, 78) = 3.17, p = .048) and the results of multiple comparisons of the three types showed that the significance probability between voice-based and tempo-based (p = .074) and between voice-based and accent-based (p = .089) were significant using a level of significance of 0.1. Thus, it seemed that voice-based vibration patterns were more comprehensible than the other two types especially when the user is allowed little opportunity to understand the correspondence between the vibration pattern and the notification message.

Our analysis of the participant opinions indicated that 24 out of 27 participants preferred the voice-based vibration pattern; however, 2 of the 27 participants preferred the tempo-based, whereas 3 of the 27 participants preferred the accent-based vibration. Those participants who preferred the voice-based pattern said that the vibration more closely reflected his/her speech than the other types.

Although the voice-based vibration pattern seemed to be effective, further analysis indicated that the number of correct answers was reduced when the vibration pattern represented a monotonous notification message. This could lead to a decrease in the comprehensibility of voice-based vibration patterns.

4 Conclusion

This study led to the development of an appropriate method for designing a vibration pattern through experimentation. We found the voice-based vibration pattern, including the characteristics of the user’s pronunciation, could be more effective than the other patterns. However, the comprehensibility of a voice-based vibration pattern could be reduced by the specific characteristics of pronunciation such as monotonous notification messages. Therefore, this problem should be addressed to improve the effectiveness of the method.