Two-Step Input Spatial Auditory BCI for Japanese Kana Characters

Conference paper
Part of the Advances in Cognitive Neurodynamics book series (ICCN)

Abstract

We present an auditory stimulus optimization and a pilot study of a two-step input speller application combined with a spatial auditory brain–computer interface (saBCI) for paralyzed users. The application has been developed for 45, out of 48 defining the full set, Japanese kana characters in a two-step input procedure setting for an easy-to-use BCI-speller interface. The user first selects the representative letter of a subset, defining the second step. In the second step, the final choice is made. At each interfacing step, the choices are classified based on the P300 event-related potential (ERP) responses captured in the EEG, as in the classic oddball paradigm. The BCI online experiment and EEG responses classification results of the pilot study confirm the effectiveness of the proposed spelling method.

Keywords

Brain–computer interface (BCI) P300 ERP EEG 

53.1 Introduction

A brain–computer interface (BCI) is a technology that uses brainwaves only to control a computer, or a machine, without any body muscle movements [1]. This technology could allow disabled people, such as those suffering from locked-in syndrome (LIS), to regain communication skills or to manage daily life support-related functions. In our research, we focus on an auditory and non-invasive BCI modality, since it allows for an application without any training even for the naive user.

Among the state-of-the-art BCIs, the visual modality is the most successful because, the evoked P300 responses usually have the largest amplitudes allowing for the easiest classification [2]. However, it is known that some LIS users (e.g., advanced stage amyotrophic lateral sclerosis patients) cannot rely on visual modalities because they gradually lose intentional muscle control, including eye movements, focusing, or intentional blinking. Our research hypothesis is that the spatial auditory BCI (saBCI) [3, 4] modality shall be a more suitable solution for establishing communication with LIS users and for creating the Japanese speller requiring a minimum of 45 characters out of 48 available for a smooth communication. In order to do so, we propose to utilize the two-step input procedure to create an easy-to-use speller interface with a limited number of choices (commands) in each step. We also test various auditory stimulus scenarios using different combinations of female and male voices in the two-step input protocol. We report results from a series of experiments in which the users were asked to spell short Japanese words using the two-step input scenarios.

The remainder of the paper is organized as follows. In the next section, we describe the online BCI experiment set-up, EEG acquisition, filtering and classification procedures, as well as the two-step input Japanese kana characters’ speller paradigm. Then, the results of the online BCI experiments are discussed together with the P300 response latencies. Finally, conclusions are drawn and future research directions outlined.

53.2 Methods

In the experiments reported in this paper, the Japanese kana character set-based BCI-speller paradigm was investigated. The paradigm was designed to extend our previously reported pilot study [4]. In order to improve the previous 25 characters-based paradigm, we designed the new interface as follows: the sound stimuli of male and the female voices were introduced in various input steps; “a step-back” command was added in the second step input using a tone of 440 Hz in order to let the user to retreat from a possible error in the first step; in the first step, the number of choices was increased from five to ten (see Fig. 53.1). In order to create the spatial sound images with stimuli coming from distinct locations known to the user, we tested the following combinations. We used the sound stimulus patterns as follows: only the female voice modality with the same vocalization presented in both the input steps; male → female voices modality in the first and second steps, respectively; female → male voices modality with an opposite set-up to the previous one.
Fig. 53.1

A diagram of the proposed two-step Japanese kana characters input procedure

The speech occurrences were adopted from a sound dataset constructed of male and female natural voice recordings available for research purposed from [5]. We used the sound stimuli of 45 Japanese kana characters. The auditory stimuli were delivered from the ten distinct spatial locations using loudspeakers positioned in two horizontal lines in front of the user’s head. The five loudspeakers were at the user ears’ level and the remaining five in a line one meter above. In order to manage the spatial sounds distributions and precise presentations to evoke the time aligned, P300 responses an in-house multimedia application which was developed in MAX environment [6]. The MAX application communicated with the in-house extended EEG acquisition, filtering, and classification environment BCI2000 [7] using an UDP network protocol. We conducted the online EEG experiments in order to verify a possibility to input the set of Japanese kana characters of the proposed spatial auditory BCI (saBCI) speller paradigm. We tested whether there were significant differences in accuracies among the proposed spatial sound stimulus patterns. In the experiments, ten healthy users took part (mean age of 24.0 years old, standard deviation of 0.63 years). The online (real-time) EEG experiments were conducted in accordance with the WMA Declaration of HelsinkiEthical Principles for Medical Research Involving Human Subjects. The procedures were approved and designed in agreement with the ethical committee guidelines of the Faculty of Engineering, Information and Systems at University of Tsukuba, Japan. Each user first conducted a short psychophysical test with a button press behavioral response to confirm understanding of the experimental set-up. Next, the EEG online BCI protocol experiments were conducted. In the EEG experiments, each user was instructed to spell a sequence of Japanese kana characters presented in random order as in the classical P300 response-based oddball paradigm [1]. The following two types of experimental sessions were performed for each user in the EEG experiments: training session—using only the five Japanese vowels (a, i, u, e, and o) in a simple single-step-spelling set-up in order to train a stepwise linear discriminant analysis (SWLDA) classifier [8] to discriminate the P300 responses (a binary classification mode of target versus nontarget brainwave evoked responses); a copy spelling test session—to test online the two-step-input Japanese kana characters speller paradigm. As a task in the test experiment, four Japanese words (ho-ta-ru, na-ma-e, fu-ne-o-mo-tu, and shi-ra-yu-ki-hi-me) were spelled by the users. During the online BCI experiments, the EEG signals were captured with eight active electrodes g.LADYbird by g.tec medical instruments GmbH, Austria. The electrodes were connected to an EEG amplifier \( \upsilon Amp \) by Brain Products GmbH, Germany. The electrodes were attached to the following head locations Cz, CPz, P3, P4, Cp5, Cp6 F3, and F4, as in the 10/10 extended international system. The ground and reference electrodes were attached at FCz and the left earlobe head locations, respectively. The captured online EEG signals were processed by the in-house enhanced BCI2000 application [9] using SWLDA classifier with features drawn from 0 to 800 ms ERP intervals. The sampling frequency was set to 500 Hz, a high-pass filter at 0.1 Hz and the low-pass filter at 40 Hz, with an electric power line interference rejection notch filter set in a band of 48–52 Hz to avoid sub-harmonic interferences. The sound stimuli were presented randomly with an inter-stimulus-interval (ISI) set to 300 ms. Each sound stimulus had a length of 300 ms either. The brainwave response signals (ERPs) in each classification step were averaged 20 times (in the classifier training sessions) and only 10 times (in the testing two-step-input spelling paradigm).

53.3 Results

The psychophysical experiments conducted with all the ten users confirmed no significant differences among the behavioral reaction times (button presses) to the used spatial stimuli. The EEG experiments conducted with the ten participating users with the proposed Japanese kana characters in two-step-input speller using spatial auditory BCI paradigm have been reported below. The obtained grand mean averaged ERP responses from the ten users have been depicted in Fig. 53.2. The P300 responses have been clearly depicted for targets (purple lines) in the ranges of 350–600 ms. The grand mean averaged result confirmed the validity of the proposed saBCI-speller paradigm settings and configuration. The clear P300 responses in comparison to the nontargets (blue lines in Fig. 53.2) validated the possibility to discriminate those ERPs using machine learning approaches and especially the proposed application of the SWLDA. The Japanese kana characters set in two-step-input spelling accuracy results of the ten participating users are summarized in Table 53.1. The scores of the users’ majority were above the two-step-input BCI-speller theoretical chance level of 1.7 %. The female, male → female and female → male stimulus modalities resulted with nonsignificant differences of means as tested with the pairwise t-tests. The ten tested saBCI-speller users reported preferences for the mixed male and female voice patterns due to more easy orientation in the two-step interface. The results of the conducted online EEG experiments have shown the feasibility of the Japanese kana characters set in two-step-input saBCI-speller paradigm. Also we confirmed that the accuracy of each modality had no significant differences. A video documenting the online saBCI-speller experiment is available at [10].
Fig. 53.2

The grand mean averaged ERP brain responses of all users participating in the reported online two-step-input BCI-speller experiments collected within the training sessions. The purple middle (mean) lines depict the ERPs to targets (note the clear P300 positive deflections after about 350 ms and lasting till about 900 ms in this study). The blue lines represent the nontarget averaged brainwaves, respectively. Normalized standard errors are depicted with light color intervals surrounding the mean responses (Color figure online)

Table 53.1

Spelling accuracies of the two-step-input Japanese characters paradigm obtained from the online BCI experiments with a theoretical chance level of 1.7 %

User number

Voice pattern

First step accuracy (%)

Final (second step) accuracy (%)

#1

Female

53.0

53.0

Male → female

23.5

5.9

Female → male

35.3

17.6

#2

Female

17.6

0.0

Male → female

11.8

5.9

Female → male

0.0

0.0

#3

Female

53.0

53.0

Male → female

76.5

47.1

Female → male

53.0

53.0

#4

Female

0.0

33.3

Male → female

23.5

17.6

Female → male

11.8

5.9

#5

Female

70.6

47.1

Male → female

76.5

17.6

Female → male

82.4

64.7

#6

Female

47.1

35.3

Male → female

35.3

23.5

Female → male

47.1

23.5

#7

Female

76.5

76.5

Male → female

70.6

53.0

Female → male

35.3

23.5

#8

Female

82.4

70.6

Male → female

94.1

82.4

Female → male

82.4

64.7

#9

Female

29.4

23.5

Male → female

47.1

23.5

Female → male

35.3

23.1

#10

Female

47.1

35.3

Male → female

53.0

41.2

Female → male

41.2

11.8

The averaged scores were 42.76, 31.77, and 28.78 %, for female, male → female and female → male voice patterns, respectively

53.4 Conclusions

In this paper, we reported the results obtained with a two-step input Japanese syllabary saBCI-speller paradigm using 45 Japanese kana characters allowing for a smooth syllabary-based communication.

The results obtained allowed the ten participating users to spell short words using two steps, as reported in Table 53.1. The mean results of all the users were above the theoretical chance level of the proposed two-step-input saBCI-speller. However, zero accuracies were also observed at sporadic cases. The preliminary yet encouraging results presented call for more research into two-step input spatial auditory BCI-speller paradigms. The next research steps will include the spatial auditory stimulus optimization for handicapped or bedridden subjects, who cannot utilize a full surround sound acoustic environment. We plan to continue this line of research in order to apply the method in the online BCI application for patients suffering from locked-in syndrome.

References

  1. 1.
    Wolpaw, J., Winter Wolpaw, E. (eds.): Brain-Computer Interfaces: Principles and Practice. Oxford University Press, Oxford (2012)Google Scholar
  2. 2.
    Chang, M., Nishikawa, N., Struzik, Z.R., Mori, K., Makino, S., Mandic, D., Rutkowski, T.M.: Comparison of P300 responses in auditory, visual and audiovisual spatial speller BCI paradigms. In: Proceedings of the Fifth International Brain-Computer Interface Meeting 2013, Asilomar Conference Center, Pacific Grove, CA USA, Graz University of Technology Publishing House, Austria (June 3–7, 2013) Article ID: 156Google Scholar
  3. 3.
    Rutkowski, T.M., Cichocki, A., Mandic, D.P.: Spatial auditory paradigms for brain computer/machine interfacing. In: International Workshop on the Principles and Applications of Spatial Hearing 2009 (IWPASH 2009)—Proceedings of the International Workshop, Miyagi-Zao Royal Hotel, Sendai, Japan (11–13 Nov 2009) P5Google Scholar
  4. 4.
    Chang, M., Mori, K., Makino, S., Rutkowski, T.M.: Spatial auditory two-step input japanese syllabary brain-computer interface speller. In: Procedia Technology, vol. 18, pp. 25–31 (2014)Google Scholar
  5. 5.
    Amano, S., Sakamoto, S., Kondo, T., Suzuki, Y.: NTT-Tohoku University familiarity controlled word lists (FW03). Speech resources consortium. National Institute of Informatics, Japan (2006)Google Scholar
  6. 6.
    http://cycling74.com/: Max 6 (2012)
  7. 7.
    Schalk, G., Mellinger, J.: A Practical Guide to Brain-Computer Interfacing with BCI2000. Springer, London (2010)CrossRefGoogle Scholar
  8. 8.
    Krusienski, D.J., Sellers, E.W., Cabestaing, F., Bayoudh, S., McFarland, D.J., Vaughan, T.M., Wolpaw, J.R.: A comparison of classification techniques for the P300 speller. J. Neural Eng. 3(4), 299 (2006)PubMedCrossRefGoogle Scholar
  9. 9.
    Matsumoto., Y., Makino, S., Mori, K., Rutkowski, T.M.: Classifying P300 responses to vowel stimuli for auditory brain-computer interface. In: Signal and Information Processing Association Annual Summit and Conference (APSIPA), 2013 Asia-Pacific, pp. 1–5 Paper ID 388 (2013)Google Scholar
  10. 10.
    Rutkowski, T.M.: Japanese kana character spatial auditory BCI speller. YouTube video available from: http://youtu.be/BY1kSqh1NEY

Copyright information

© Springer Science+Business Media Singapore 2016

Authors and Affiliations

  1. 1.Life Science Center of TARAUniversity of TsukubaTsukubaJapan

Personalised recommendations