Keywords

1 Introduction

Members of the Deaf and Hard of Hearing (DHH) community find it difficult to perceive rhythms, patterns, sound loudness, and other sound characteristics. Therefore, it is difficult for DHH persons to regulate the loudness of their voice in speech practice and to match the rhythm of others during dance practice. Furthermore, in movie theaters, the DHH can only enjoy content based on visual information such as text. In sports venues, they may find it difficult to understand the atmosphere in terms of the cheering and excitement. Their sense of presence and unity may therefore be greatly inhibited when simply mimicking the enjoyment of non-DHH persons. The authors learned sign language through connections with the DHH while at university. Moreover, they worked as sign language interpretation volunteers and established sign language circles while managing an NPO along with the DHH community. In 2014, based on the use of technology to convey sound to the DHH, the authors began research on the “Ontenna” user interface, which enables sound to be felt by the body [1] (Fig. 1).

Fig. 1.
figure 1

Ontenna is a user interface that enables the user to hear sound through his/her body via tactile feedback. Ontenna is worn on the hair of the person in the center of this figure and on the collar of the person on the right-hand side of this figure.

Full size image

2 Related Work

In numerous studies, sound characteristics have been converted into visual and tactile information that is then presented to the DHH [2,3,4,5,6,7,8]. However, visual feedback increases the visual burden when visually transmitting sound information to the DHH, whose daily lives rely on visual information. Tactile feedback may inhibit movements of hands, arms, or feet, thereby placing a burden on the user when communicating in sign language or when performing physical actions. The ideas and opinions of the DHH were extracted through inclusive design techniques and repeated quick-prototyping using 3D printers, laser cutters, Arduino, and other digital fabrication technologies to design the optimal user interface for the DHH.

3 Study of the System Design

By obtaining real impressions from the use of prototypes, the DHH indicated aspects to be changed or improved. Table 1 summarizes the characteristics of each prototype, prototype users, interview period and frequency, opinions resulting from prototype implementation via managers and users, and considerations for improving upon this prototype.

Table 1. Changes in the Ontenna prototypes. We summarize the prototype evolution over four years from 2014 to 2017 in chronological order. Each prototype is presented with the year of development, image, number, description, characteristics, prototype testers, interview period and frequency, prototype implementation manager, user opinion, and aspects to consider for improving upon the prototype.
Full size table

The fabricated devices were loaned for a period of approximately one month to schools for the deaf in three locations across Japan that cooperated in the experiment to test these systems in real situations (Fig. 2). Concerns were raised pertaining to the difficulty of operation, inconvenience of charging, and weakness of the clip component. Furthermore, the students provided their opinions, stating the following: “I don’t want to attach it to my hair because I have a cochlear implant,” “It would be easier if it was attached to the earlobe,” and “I would like to attach it to my collar.” This feedback indicates differences in the preferred attachment method depending on the user. Accordingly, the system design was improved by changing the clip type to make it easier to attach to clothes, arms, and earlobes, among other locations.

Fig. 2.
figure 2

Demonstration at a deaf school.

Full size image

4 Final Product

Ontenna is attached to the hair, earlobe, collar, or sleeve, and it has a user interface for physically sensing sound characteristics via vibrations and light. Sound characteristics transmitted after sensing sound pressures between 60 and 90 dB are converted to 256-stage vibrations and light intensities. The cadence patterns of the sound sources are converted in real-time such that sound rhythms, patterns, and volume can be perceived (see the left-hand side of Fig. 3).

The controller can be used to control multiple Ontenna devices at the same time via the communication function. Because it uses a 920-MHz-band radio wave, there is little interference, and any number of Ontennas can be controlled within a 50 m radius (see the right-hand side of Fig. 3). In addition, it is possible to transmit sound information via AUX or MIC connections. The specifications of this system are presented in Table 2.

In June 2019, Ontenna was released as a product. Furthermore, Ontenna was freely distributed to 88 of 102 schools for the DHH in Japan (as of February 2022).

Fig. 3.
figure 3

Left: Ontenna; Right: Controller

Full size image
Table 2. Specifications
Full size table

5 Usage Examples at Schools for the Deaf

Teachers at a school for the deaf asked students to trigger the LEDs after showing how Ontenna lit up when the teacher spoke. Accordingly, the students spoke more proactively than before such that Ontenna would light up, and they were able to control the volume of their voices. Moreover, they were able to understand that the strength of the light and vibrations changed according to the volume of the voice, and they modulated their voices to gradually become louder or quieter (Fig. 4). In addition, in a music class, Ontenna enabled students to adjust the strength of their breath while playing recorders. In a dance class, the controller enabled multiple students to synchronize their movements. The teachers provided the following feedback: “Rhythm could be transmitted to students from far away rather than by tapping them on the back,” “ During drum performances, it was easier to match the rhythms by showing the rhythm to multiple students simultaneously,” and “It was easier to teach because individual students were accurately shown the rhythm”.

Fig. 4.
figure 4

Left: voice training using Ontenna; center: recorder practice with Ontenna; right: dance practice with Ontenna

Full size image

6 Usage Examples in Entertainment

Ontenna is also used in the field of entertainment. For example, when Ontenna was used in a soccer game, members of the DHH community commented, “I understood the rhythm of the chanting nearby,” and “The silent tension that filled the stadium during a penalty kick was conveyed”. When watching Kyogen, a traditional Japanese performance art, DHH patrons said, “I felt the vocal intonations” and “I could even sense the onstage footsteps”. At a tap dance event, DHH patrons commented, “I was able to experience the rhythm of the tap dance in my body,” and “It was fun because the vibrations and lights matched the dancing”. Furthermore, the participants with normal hearing ability stated the following: “I had a strong sense of the ambience because of the vibrations” and “It was a lot of fun to experience the sense of oneness with the lights”. After attending a ping-pong game, the DHH participants commented, “It was fun to experience the rhythm of the rally sounds” and “I could sense the different sound rhythms such as when it hit the net and during serves.“ Furthermore, those with normal hearing who also attended said the following: “It felt more real because of the sense of touch as well as the visuals and audio” and “I got a stronger sense of rhythms with smaller sounds.” The system configurations for each event are illustrated in Fig. 5.

Fig. 5.
figure 5

Top left: watching a soccer game using Ontenna; top right: watching Kyogen using Ontenna; bottom left: watching tap dancing using Ontenna; bottom right: watching ping pong using Ontenna

Full size image

7 Development and Release of the Ontenna Programming Education Environment

In December 2020, we released a programming environment using Ontenna, which has since been installed in over 80% of schools for the deaf in Japan. Using Scratch (https://scratch.mit.edu/), a visual programming tool, it is possible to change the color of the LEDs and the strength of the motors in Ontenna devices. We created an environment in which children with hearing impairments can overcome the obstacles they face at school by programming Ontenna themselves (Fig. 6). A questionnaire survey was conducted for seven classes in four schools where these systems were introduced. The responses were obtained using a 10-point scale, with 10 being the best and 1 being the worst. The results indicated the usefulness of the system (Fig. 7). In addition, the costs associated with internet-of-things (IoT) programming education, which is difficult to introduce due to financial constraints, were reduced by using Ontenna and PCs that were already present in the schools. In addition, by distributing instructional plans, lesson slides, and worksheets free-of-charge, we were able to reduce the learning burden of teachers.

Fig. 6.
figure 6

Left: Ontenna programming environment; right: a deaf student programming their own ideas into Ontenna

Full size image
Fig. 7.
figure 7

Results of the questionnaire survey

Full size image

8 Conclusion

In this report, we described the development of the Ontenna user interface for physically experiencing sound, which was researched and developed in cooperation with the DHH community. Case studies involving the use of Ontenna at schools for the deaf and in the field of entertainment were also presented. Ontenna was finalized as a product in June 2019, and it has been used for music and physical education lessons at schools for the deaf across Japan. In particular, changes were observed in speech and rhythm education applications upon the implementation of this system, where students who were not previously aware of sounds began to become interested in music and those who had difficulties understanding rhythms were able to accurately mimic rhythms. In the field of entertainment, including movies, traditional arts, and sports, Ontenna has been used to create novel spectator experiences for the DHH community. Furthermore, apart from DHH persons, Ontenna has been found to have potential use in creating added value for those with normal hearing in terms of their sense of realism and integration into an event. In addition, we developed a function that allows Ontenna to be programmed and made the educational environment available to schools for the deaf to promote programming education.