Keywords

1 Introduction

Hearables may be the wearable of the future. They fit on or in the ear, providing audio playback, soundscape argumentation, and integrate biometric sensors [6]. More than $28 million have been raised from crowdfunding for hearables since 2014 [5] showing an increased interest in hearables. However, many start-ups have struggled to deliver, and have been forced out of the market in the process. Nick Hunn projects that the market for hearables within 2 years will increase to more than 230 million units, with a market revenue of more than $30 billion [5].

Hearing instruments are a medical device subcategory of hearables, which offer advanced capabilities for augmenting listening scenarios, including amplification, noise reduction and speech enhancement. The latest generation of hearing instruments connects to smartphones through Bluetooth, enabling them to communicate with other apps or cloud services supporting the IFTTT standard, effectively making them IoT connected devices.

Hearing instruments primarily support enhanced speech intelligibility in challenging listening scenarios characterized by speech in noise or multiple talkers. However, only a small fraction of the 360 million people suffering from severe hearing loss [12], including 48 million Americans (20% of the populatioon) [9] suffering from hearing loss, use hearing instruments.

In a previous study, Laplante-Levesque et al. [7] investigated the usage of hearing instruments, and compared self-reported use, and historical summarized use from the hearing instrument (average on/off time). It was found that there are two distinct types of behaviors associated with hearing instrument usage. Users wearing the device from morning to bed, and users using the hearing instruments when needed. The hypothesis of this study is that each participant have a unique behavior, and that there may be more than one usage pattern. They furthermore concluded that the average wear of a hearing instrument averaged 10.5 h. This is well beyond the battery capacity of current hearables, with Apple AirPods claiming up to 5 h play on a charge [1] and technologies with binaural microphones, such as the Doppler Labs Here One and the Bragi Dash claims 3 h of use on a battery charge [3, 4]. In comparison, current hearing instruments batteries can sustain a week of use, or more, before the need for changing batteries.

This paper investigates the usage patterns of hearing instrument users based on user initiated program and volume changes through a pilot study of 7 weeks. These adjustments are converted into time series data saved in the cloud using IFTTT to transfer data. Previous studies have primarily used summarized historical data retrieved from the hearing instrument software, whereas IoT devices may potentially learn from usage data, such as volume and program interactions, to dynamically adapt the hearing instruments to behavioral patterns. In this article hearing instruments will also be referred to as hearables.

2 Method

6 participants (median age 61.8) with more than 5 years experience of hearables were recruited for the study. Half of the participants were retired, while the other half are still working. Participants were equipped with two Oticon Opn™ hearing instruments connected personal iPhones using Bluetooth. All user initiated program selection or volume changes were recorded as time series data stored over a 7-week period. All participants were provided with a Google Drive account used for data collection, allowing them to retain full ownership of the data. The hearing instruments were fitted based on audiograms by an audiologist to provide individualized frequency dependent amplification for each subject. Rather than a single optimized setting the hearables were fitted with four alternative programs from the Oticon OpenSound Navigator™.

These programs are trade-offs between speech and noise balance, i.e., speech intelligibility, and of background sound amplification. The OpenSound Navigator works with three modules to analyze the sound, these are described by Le Goff et al. [8] as: Analyze, analyzes the sound environment both omnidirectional, and backward, estimating where a noise sources are placed. This simulates how sound normally are perceived by the human ear, with more sound attenuation from the back and the sides of the listener. Balance, which determines speech sources and attenuate noise sources between speech sources. This balances the signal-to-noise ratio (SNR). And, noise removal, which attenuate noise sources and amplifies speech above the hearing threshold.

Each of the programs gives various support depending on the context, from simple environments such as speech in quiet to more complex environments with multiple talkers and ambient background noise, such as an outdoor cafe.

The four programs are:

  • P1: Resembling an omnidirectional perception with a frontal focus. Sounds from the sides and behind the listener are slightly suppressed to resemble the dampening effect of the pinna.

  • P2: similar to P1 but gently increasing balance and noise removal when encountering complex listening environments.

  • P3: similar to P1 but increasing balance and noise removal even in simple listening environments.

  • P4: similar to P3 with high sensitivity to noise increasing balance and noise removal in all listening environments.

2.1 Participants

6 participants were recruited for the study (6 men). The median age was 61.8 years (std. 11.1 years). All participants have used hearables for more than 5 years. All have an iPhone 4S or newer. Half of the participants are retired, and the other half are working. The hearing loss ranges from mild (26–40 dB), moderate (41–60 dB) and severe (61–80) as described by the WHO [11]. Two participants were not included in the study due to lack of data or missing data. A short summary of each subject is provided in Table 1.

Table 1. Demographic information related to 4 subjects
Full size table

The study was carried out in Denmark in the autumn of 2016, and follow up in January and February 2017. Participants were instructed at Eriksholm Research Centre.

2.2 Apparatus

Each participant were equipped with two Oticon Opn™ hearing instruments, stereo Bluetooth low energy (BLE) 2.4 GHz, Near-Field Magnetic Induction (NFMI). All participants used (personal) iPhone 4S or newer models, Bluetooth 4.0 (or newer). The data streamed by the hearables consist of any user initiated program change or volume changes (\(-4\) to 8) accompanied with a time-stamp of the interaction, stored in the cloud on a test subject owned Google spreadsheet and shared via Google Drive. The hearing aids were fitted with four audio profiles P1, P2, P3 and P4, described earlier.

The participants were provided with a private test user Google account prior to the experiment. The account was used for data collection, and the participants have full ownership of the account and data. Data was collected over a 7-week period.

2.3 Procedure

Participants were fitted with OPN hearing instruments. The hearing instruments were fitted based on a unique frequency dependent volume amplification for each subject, by an audiologist. User initiated program and volume changes are collected trough the ON app, which in combination with the IFTTT app collects and store usage patterns as time series data. Each user initiated action is stored as a row on a private Google drive spreadsheet. 10 IFTTT recipesFootnote 1 were installed on the participants smartphone. The participants were encouraged to explore the hearables and their functionality with no further instruction provided in which scenarios the programs would be best suited. Participants could then test the device, while the researcher and an audiologist were present. The participants were informed that data would be continuously streamed for the duration of the experiment. Each participant was fitted with four programs, through the Genie 2.0™ fitting software using the OpenSound Navigator. Follow up consultations with an audiologist was planned for the end of the study. These consultations included an interview about the use of the hearables along with: Usage history collected by the device compared with the collected cloud data. Secondly, inquiring into the usage of specific programs to further understand the users preferences and intents in various scenarios. Leading to defining new program settings for a follow up study. The aim would be to tease apart the need for increasing attenuation of ambient sound sources, noise removal and improving speech intelligibility associated with different scenarios.

3 Weekday Program Usage over 24 Hours

The program patterns in Figs. 1 and 2 ranging from P1 (beige), P2 (brown), P3 high (light blue), to P4 (dark blue), illustrate the large differences between users, their contrasting needs throughout the day, as well as their changing preferences for weekday vs weekend activities. General trends towards increased support during the day can be seen for users 1, 5 and 7. Conversely, less need of support in the evening is reflected in the behavior of user 2. In addition, the bright sound represented by the P2 (brown) versus the full sound of P3 (light blue) may indicate how preferences for the P2 increases speech intelligibility, whereas P3 provides a less intense listening experience. Likewise, the program usage on weekdays could be driven by the demands of work related activities, while the preferences on weekends might to a larger degree reflect individual baselines defining their cognitive processing needs [2, 10].

Fig. 1.
figure 1

Aggregated average program time. The time is displayed in minutes for each hour, and is aggregated for the full data collection period. The use of P1 (beige), P2 (brown), P3 (light blue), and P4 (dark blue), varies for each test subject as well as over the course of the day. (Color figure online)

Full size image

4 Changing Preferences in the Weekends

4.1 Weekends as a Baseline

In Fig. 2a comparison between weekday usage (left side of the figure) and weekends (right side of the figure) is illustrated for subject 2 and 5. It can immediately be noticed that the behavior pattern varies from weekdays compared to weekends. A clear trend of preferring P1 in the weekend is evident. The preference for a more natural sound in the weekend can be due to a less challenging context, compared to weekdays (and working days). It can also be observed that the weekends have a later onset of the day.

From these observations it seems as the weekend reflects a baseline state where the user prefers natural sound and does not need the enhanced speech intelligibility and noise reduction associated with the P4 program.

Fig. 2.
figure 2

Comparison of weekday and weekend patterns for subject 2 and 5. The data is aggregated over the full study period, and is displayed as an average minute per hour. Notice the distinct pattern of less support in the weekends (brown and teal colors are preferred). P1 (beige), P2 (brown), P3 (light blue), and P4 (dark blue). (Color figure online)

Full size image

4.2 Varying Context Creates Different Needs

An interesting observation from Fig. 2a and b for subject two, is the distinct pattern of removing background noise from morning to late afternoon. In a follow up interview, this subject indicated that he works in the transportation industry, and indeed works between 8AM and 4PM. The choice of this program is to reduce noise. This subject along others, indicated that the weekends have the least troublesome scenarios, and a more natural sound, such as the one provided by P1, is preferable in these contexts.

Subject 7 have a distinct pattern using the automatic and supportive programs, especially P3. These programs increase speech intelligibility and have a higher sensitivity to background noise. This subject play cards 2–3 times a week for several hours. Due to the nature of the card game and a room with poor acoustics, the P2 and P3 program increases speech inteligibility.

Both of these subjects mentioned that the weekends contains less challenging scenarios. Anecdotal, the reason for wearing the hearables later in the day is caused by reading the newspaper in the morning. The newspaper creates an uncomfortable sound environment containing rattling and sharp noises, where a quiet environment is preferred.

Fig. 3.
figure 3

Preference of using the hearing aid over time. The data is aggregated and averaged per week of collected data. Notice how the first weeks include use of more programs, while this decline towards the end of the data collection period. This indicates that the user finds a “preferred” setting over time. Some subjects have missing data due to lack of Internet connection (outdoor environments).

Full size image

5 Program Use over Several Weeks of Use

From Fig. 3 the preferences for program use over several weeks can be observed. Due to some weeks without data, caused by a lack of Internet connection (e.g., in outdoor environments), some subjects have fewer weeks represented than others. It can be observed that the majority of the subjects uses two or more programs the first 3 weeks. While at the end of the pilot study they seem to prefer two programs, typically P1 and a program that assist in challenging listening environments. This indicates that over time the participants become aware of the capabilities of the hearables, in which scenarios it can support them as needed, and at which times it performs the best. From the figure it is visible that a preference for the more open and natural sounding P1 is used most frequently. This indicates that the participants prefers a natural sound, and when a challenging scenario occurs, they change to a supportive program.

A second observation indicates that the preference between the changes in many cases includes two contrasting programs. Over time a preferred supportive program for the subject emerges.

5.1 Perceived Sound Quality

The perceived sound quality is a motivator for behavioral use of hearables. The primary focus from the established hearing aid industry have been on increased speech intelligibility and dealing with challenging listening scenarios. However, from interviews of the subjects in this study, the majority of the wear time is not spent in challenging environment. The natural open characteristics of P1 seems to provide a natural sound environment, which provide sufficient amplification in most listening scenarios, involving only few speakers and less background noise. As confirmed by accumulated usage history, the P1 is used to reproduce a natural sound up to 75% of the time.

6 Program Duration and Volume Changes

The program changes can explain part of the behavioral patterns of each of the subjects. The programs can be observed as macro settings modifying a soundscape by adjusting the noise removal and attenuation of ambient sound sources. As earlier mentioned, P1 has the least effect on the soundscape, with a frontal focused omnidirectional producing a natural sound, while P4 has increased noise removal and attenuation of ambient sound sources. The interaction between programs and volume can be interpreted as user intents.

The volume control on the other hand works as a micro adjustment. By controlling the volume gain the user can zoom in or out of a soundscape, alternating how present in the current context they wish to be. This does not affect the reproduced sound from the programs, only the gain and intensity of the reproduced sound.

6.1 Fine-Tuning Using the Volume Control

To illustrate the use of the volume control for fine-tuning, the usage patterns for subject 2 and subject 7 can be observed in Fig. 4. In Fig. 4 the average change in volume gain is displayed, with respect to the two contrasting programs of P1 and P4, blue for decreasing and orange for increasing gain.

Figure 4a indicates a unique pattern for subject 2 of a need for an increase in volume, around meal times. In the weekend, shown in Fig. 4b the volume is primarily decreased, and only increased in the late evenings on weekends. This pattern is contrasting with subject 7 s pattern, seen in Fig. 4c where the volume is always decreased in P1. In P4 there is a contrasting volume change from evening meal time, and just after this meal time.

Fig. 4.
figure 4

Comparison of volume interactions with respect to weekdays (left figures) and weekends (right figures) for subject 2 (top) and subject 7 (bottom). Notice the distinct difference in volume patterns between the two subjects. Observe the contrasting volume changes for weekdays versus weekends. (Color figure online)

Full size image

Comparing just these two programs for two subjects with respect to volume shows how the subject intentionally uses a combination of a program and a volume to adjust the auditory experience. Furthermore, it highlights the difference between usage pattern between two subjects. One prefers to primarily increase volume, while the other prefers to decrease volume. These changes also occurs at different time intervals, indicating a need for personalized hearables.

7 Conclusion

These results show how user generated volume and program interaction data may capture preferences for personalizing the listening experience to the changing context. The usage patterns highlight individual needs for selecting contrasting programs rather than a medium one size fits all setting often provided by default. The shared user generated data might potentially be used to learn behavioral patterns enabling the devices to automatically adapt their settings and thus optimize the user experience of hearables.

It seems that at least two programs are needed to optimize the hearing experience. Test subjects prefer to change settings of the hearables in the course of a day. This is visible in the emerging patterns, where each user has unique usage patterns. These patterns are influenced by the changing context.

At least two programs are needed to satisfy the needs of the users of hearables. It can be observed that most users tend to have an early onset of testing the various modification of the soundscape observed by changing programs. Later in the period they find a preferred program that works in most situations. For all subjects this is program P1, the one that reproduces sound most naturally.

These observations could be the foundation for the future design of hearables. The findings in this paper can be used to optimize, not only the listening experience, but also how the devices can learn from human behavior to adapt to the user. This could lead to a “I forgot I’m wearing an in-ear device”, which reproduces sound naturally. At the same time, the device could be used to enhance a social interaction, when needed, by enhancing speech intelligibility.

We suggest a need for better control, or smarter devices, that learns and adapts to the users individual patterns are needed in the future. These devices can be used in any hearable augmenting sound, to create an enhanced user experience.