Abstract
The gaze-independent brain-computer interface (BCI) based on rapid serial visual presentation (RSVP) is an extension of the oddball paradigm, and can facilitate communication for people with severe neuromuscular disorders. Some studies suggested that a face with eyes only (without other facial features) could evoke ERPs as high as a complete face. To evaluate the performance of a BCI system, an online system is needed. In this paper, we compared two types of stimuli: a dummy face with eyes only and a colored circle. Ten healthy subjects (8 male, aged 24–28 years, mean 26 ± 1.5) participated in our experiment. The results showed that the dummy face with eyes only had substantial advantages in online classification accuracy (t = 2.3, p = 0.04) and information transfer rate (t = 3.4, p = 0.003). Results from users’ feedback also showed that the dummy face with eyes only stimulus was more easily accepted (t = 4.12, p < 0.001).
Keywords
1 Introduction
Brain-computer interfaces (BCI) create a new link between the brain and an external device, which provides an alternative means of device control for people with severe paralysis [1–3]. The P300 is an endogenous component of the event-related potential (ERP) that reflects cognitive process such as attention and working memory [4]. With some paradigms, P300 differences can be elicited without shifting gaze, because attention can be oriented separately from eye movement [5]. This is the reason that the classical speller is considered independent [6]. However, Treder and Blankertz (2010) addressed this question by conducting experiments in both overt and covert attention conditions [7]. Brunner et al. (2010) also indicated that successful operation of the classical P300 speller substantially depends on gaze direction. However, the accuracies of the subjects were also very low [8]. Acqualagna et al. introduced and evaluated a gaze-independent BCI based on RSVP with overt attention and feature attention conditions [9]. The RSVP speller is suitable for patients with deterioration of oculomotor control. Recently, Daly et al. reported that the gaze-independent BCIs based on RSVP used dummy face pictures as stimuli obtained high performance [10]. However, in the research of gaze-independent BCIs based on RSVP, the users’ fatigue had not been taken seriously. Therefore, an improved paradigm with assessment of subjective user factors was needed, which could improve the performance of BCIs and make users more relaxed. Itier et al. reported that face-specific effects were mediated by the eye region when faces were used as stimuli [11]. That is, compared with mouths and noses, the eyes play more important roles when human faces are as stimuli in BCIs. So in terms of evoking distinct ERPs for BCIs, complete faces have no decisive advantage over faces stimuli with eyes only region [12].
In this paper, to further assess the performance of the online gaze-independent BCI system with the dummy face with eyes only approach, we explored two paradigms. These were called the “colored circle pattern (CCP)” paradigm and “dummy face with eyes only pattern (DFP)” paradigm. We assessed online and offline performance, including subjective measures (via questionnaires) as well as objective results. Two types of stimuli were colored circle and dummy faces with eyes only region, respectively.
2 Materials and Methods
2.1 Subjects
Ten healthy subjects (8 male, aged 24–28 years, mean 26 ± 1.5) participated in this study, and are designated P1 … P10. All subjects signed a written consent form prior to this experiment and were paid for their participation.
2.2 Stimuli and Procedure
Before recording began, subjects were prepared for EEG recording. We recorded from 14 EEG electrode positions based on the extended International 10–20 system [13–15]. These electrodes were Cz, Pz, Oz, Fz, F3, F4, C3, C4, P3, P4, P7, P8, PO7 and PO8. The right mastoid electrode was used as the reference and the front electrode (FPz) was used as the ground. EEG signals were recorded with a g.USBamp and a g.EEGcap (Guger Technologies, Graz, Austria) with a sensitivity of 100 µV, band pass filtered between 0.1 and 30 Hz [16–18].
After electrode preparation, we explained the task to the subjects. Their task was to silently count each time a specific “target” stimulus flashed. We explained that all stimuli would flash sequentially, always in the center of the monitor, and that subjects would sometimes have breaks during which the flashes stopped (see Fig. 2). During these breaks, subjects would be asked to report the number of flashes, and then the procedure would repeat with a new target (Fig. 1).
This study used had two conditions, which differed only in the stimuli used. One condition used the colored circle paradigm (CCP), while the other condition used the dummy face paradigm (DFP). Each condition used six different stimuli (see Fig. 2). The CCP paradigm used colored circles, while the DFP condition used cartoon faces. All twelve of these stimuli were the same size and brightness, and were developed through Photoshop.
Throughout this paper, the term ‘flash’ refers to each brief stimulus presentation. Stimuli were flashed for 200 ms, with a 100 ms delay between flashes (see Fig. 2). Thus, each trial (meaning a sequence of six flashes) lasted 1.8 s. A trial block refers to a group of trials with the same target. During offline testing, there were 16 trials per trial block and each run consisted of 5 trial blocks, each of which used a different target. During online testing, the number of trials per trial block was adaptive, as described below.
Prior to each trial block, one of the six stimuli used in that condition was presented in the center of the screen for 3 s, and subjects were told that this stimulus was the new target. Subjects had a 5 min break after each offline run. We first completed two offline runs, and then used the data to train two classifiers (one for each condition). Subjects then had a 3 min break, followed by an online experiment for one condition, then 5 min break, and then an online experiment for another condition. The order of the conditions was determined pseudorandomly. Subjects attempted to identify 24 targets during online testing.
2.3 Data Processing
The first 800 ms of EEG after each flash was used for feature extraction. A pre-stimulus interval of 100 ms was used for baseline correction of single trials. The raw feature matrix is 14 × 204 for each single flash, since there were 14 channels and 204 samples per trial. Raw features from each channel were down-sampled from 256 Hz to 64 Hz by selecting every fourth sample from the filtered EEG. The size of the resulting feature vector was 14 × 51 (14 channels by 51 time points). No special signal pre-processing in addition to conventional filtering. We performed a third order Butterworth filter (1–30 Hz). Classification relied on Bayesian linear discriminant analysis (BLDA). BLDA is an extension of Fisher’s linear discriminant analysis (FLDA) that avoids over fitting and possibly noisy datasets. The detail of the algorithm can be found in [15]. The online runs used an adaptive strategy with a variable number of trials per average [18].
2.4 Subjective Report
After completing the last online run, each subject was asked two questions (in Chinese) about each of the two conditions. These questions could be answered on a 1–5 scale indicating strong disagreement, moderate disagreement, neutrality, moderate agreement, or strong agreement. The two questions were:
-
1.
Did this paradigm make you tired?
-
2.
Was this paradigm hard?
3 Results
3.1 Objective Results - ERPs
Figure 3 shows the amplitude differences for the target vs. non-target comparison for each of the ten subjects across the three peaks studied in both conditions. These differences are presented at the electrode sites selected for statistical analysis. Electrode site P7 was selected for the N200 analysis. Electrode site Pz was selected for the P300 analysis because it is commonly largest at Pz. Electrode site Cz was selected for the N400 analysis because it typically contains the largest N400. The peak values of the difference ERPs (target minus non-target) with specific time windows were compared between the two conditions over all subjects by using paired samples t-tests. The time windows were: N200 (150 to 280 ms); P300 (280 to 450 ms), and N400 (450 to 710 ms). At site P7, N200 amplitude (target minus non-target) showed no statistically significant difference between the two conditions (t = 0.62, p = 0.54). At site Pz, P300 amplitude (target minus non-target) was significantly larger (more positive) for DFP compared to CCP (t = 2.27, p = 0.04). At electrode Cz, the N400 amplitude (target minus non-target) was significantly larger (more negative) for DFP compared to CCP (t = 2.29, p = 0.03).
Table 1 shows online classification accuracy, information transfer rate, and number of trials per average for each condition. Paired samples t-tests were used to examine differences between the DFP and CCP. Classification accuracy and information transfer rate were significantly higher for the DFP condition than the CCP condition (t = 2.3, p = 0.03 and t = 3.4, p = 0.003 respectively). The DFP condition required significantly fewer trials per average than the CCP condition (t = 2.4, p = 0.04).
3.2 Objective Results – BCI Performance
Figure 4 shows the mean classification accuracies based on offline data from each subjects using 15-fold cross-validation. The mean classification accuracy was calculated based on single-trial (not averaged) data. This analysis showed that the DFP is higher than the CCP in classification accuracy (t = 3.86, p < 0.001). The overall mean classification accuracies of all subjects based on single-trial data were 73.5 % ± 10 % (DFP) and 51.6 % ± 12 % (CCP).
3.3 Subjective Results – Questionnaire Replies
Table 2 shows subjects’ responses to the two questions. A paired-samples t-test was used to examine mean differences between the DFP and CCP. Subjects reported lower fatigue in the DFP condition than in the CCP condition (t = 4.98, p < 0.001). Subjects also reported that the DFP condition was less difficult than the CCP condition (t = 4.12, p < 0.001).
4 Discussions
For gaze-independent BCIs based on RSVP, users’ subjective report had not been fully considered. Therefore, we assessed users’ fatigue and perceived task difficulty, as well as objective performance. Itier et al. had proved that face elicited high event related potential were mediated by the eye region [10]. That is, the eyes may be central to what makes faces so special [10]. We suggested that the dummy face with eyes only was better suited than a completed dummy face for a gaze-independent BCI based on RSVP, since the dummy face with eyes only approach could be simpler and less distracting. However, we had not explored explore this approach online.
The offline results showed that, relative to CCP, DFP elicited a higher P300 (target minus non-target, p < 0.05) at Pz and N400 (target minus non-target, p < 0.05) at Cz. The DFP had no significant advantage in terms of the N200 (target minus non-target, p > 0.05) at P7 compared to the CCP pattern. The DFP had an advantage over CCP in terms of offline classification accuracies (p < 0.05). The online results showed that the online classification accuracies (p < 0.05) and information transfer rates (p < 0.05) of DFP were better than CCP. We also explored the number of trials required per selection in the online condition, and DFP required fewer trials per average (p < 0.05).
The main goal of this study was to identify the key characteristics of stimuli that can elicit large ERP differences for a BCI. However, the results are also interesting in their implications for human face processing. The two paradigms are not dramatically different in terms of the images presented, differing only in the inclusion of cartoon eyes in the DFP condition. Nonetheless, this small difference was enough to elicit significantly different ERP activity. Hence, this study supports the studies cites above, and numerous others, that show that face processing is a very important component of human cognition. The results also show that people are very good at recognizing abstract representations of eyes. We did not assess more realistic images of eyes (such as photographs), but even cartoon images such as those presented here are immediately and effortlessly recognizable.
5 Conclusions
An online gaze-independent BCI system with the “dummy face with eyes only” paradigm was presented in this paper and compared to a “colored circle” paradigm. The online results demonstrate that this new approach with dummy faces is viable as stimulus in a gaze-independent BCI based on RSVP. In addition to outperforming the colored circle paradigm in terms of objective measures, the dummy faces approach was also superior in terms of subjective report.
References
Wolpaw, J.R., Wolpaw, E.W.: Brain-Computer Interface: Principle and Practice. Oxford University Press, Oxford (2012)
Luck, S.T.: An Introduction to the Event-related Potential Technique (Cognitive Neuroscience). MIT Press, Cambridge (2005)
Birbaumer, N.: Brain–computer-interface research: coming of age. Clin. Neurophysiol. 117, 479–483 (2006)
Liu, Y., Zhou, Z.T., Hu, D.W.: Gaze independent brain–computer speller with covert visual search tasks. Clin. Neurophysiol. 122, 1127–1136 (2011)
Posner, M.I.: Orienting of attention: then and now. Q. J. Exp. Psychol. (Hove) 30, 1–12 (2014)
Wolpaw, J.R., Birbaumer, N., McFarland, D.J., Pfurtscheller, G., Vaughan, T.M.: Brain-computer interfaces for communication and control. Clin. Neurophysiol. 113, 767–791 (2002)
Treder, M.S., Blankertz, B.: (C)overt attention and visual speller design in an ERP-based brain-computer interface. Behav. Brain Funct. 6, 28 (2010)
Brunner, P., Joshi, S., Briskin, S., Wolpaw, J.R., Bischof, H., Schalk, G.: Does the ‘P300’ speller depend on eye gaze. J. Neural Eng. 7, 056013 (2010)
Acqualagna L., Blankertz B.: A gaze independent spelling based on rapid serial visual presentation. In: Proceedings of the IEEE EMBS, pp. 4560–4563 (2011)
Itier, R.J., Alain, C., Sedore, K., McIntosh, A.R.: Early face processing specificity: it’s in the eyes. J. Cogn. Neurosci. 19, 1815–1826 (2007)
Chen, L., Jin, J., Zhang, Y., Wang, X., Cichocki, A.: A survey of dummy face and human face stimuli used in BCI paradigm. J. Neurosci. Meth. 239, 18–27 (2015)
Jin, J., Daly, I., Zhang, Y., Wang, X.Y., Cichocki, A.: An optimized ERP brain–computer interface based on facial expression changes. J. Neural Eng. 11, 036004 (2014)
Kremláček, J., Kuba, M., Kubová, Z., Langrová, J.: Visual mismatch negativity elicited by magnocellular system activation. Vision. Res. 46, 485–490 (2006)
Stefanics, G., Csukly, G., Komlósi, S., Czobor, P., Czigler, I.: Processing of unattended facial emotions: a visual mismatch negativity study. NeuroImage 59, 3042–3049 (2012)
Hoffmann, U., Vesin, J.M., Ebrahimi, T., Diserens, K.: An efficient P300-based brain-computer interface for disabled subjects. J. Neurosci. Meth. 167, 115–125 (2008)
Kimura, M., Katayama, J., Murohashi, H.: Attention switching function of memory-comparison-based change detection system in the visual modality. Int. J. Psychophysiol. 67, 101–113 (2008)
Czigler, I.: Visual mismatch negativity and categorization. Brain Topogr. 27, 590–598 (2014)
Jin, J., Allison, B.Z., Sellers, E.W., Brunner, C., Horki, P., Wang, X., Neuper, C.: An adaptive P300-based control system. J. Neural Eng. 8, 036006 (2011)
Acknowledgments
This work was supported in part by the Grant National Natural Science Foundation of China, under Grant Numbers 61573142,61203127, 91420302, 61305028 and supported part by Shanghai Leading Academic Discipline Project, Project Number: B504. This work was also supported by the Fundamental Research Funds for the Central Universities (WG1414005, WH1314023 and WH1516018).
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer International Publishing Switzerland
About this paper
Cite this paper
Chen, L., Allison, B.Z., Zhang, Y., Wang, X., Jin, J. (2016). An Online Gaze-Independent BCI System Used Dummy Face with Eyes Only Region as Stimulus. In: Schmorrow, D., Fidopiastis, C. (eds) Foundations of Augmented Cognition: Neuroergonomics and Operational Neuroscience. AC 2016. Lecture Notes in Computer Science(), vol 9743. Springer, Cham. https://doi.org/10.1007/978-3-319-39955-3_3
Download citation
DOI: https://doi.org/10.1007/978-3-319-39955-3_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-39954-6
Online ISBN: 978-3-319-39955-3
eBook Packages: Computer ScienceComputer Science (R0)