Keywords

1 Introduction

The quintessential application for Brain-Neural Machine/Computer Interface (BNCI) [1, 2] has been as an assistive technology for individuals suffering from neural dysfunction of such severity that other assistive technologies cannot offer appropriate functionality. Relevant conditions have included amyotrophic lateral sclerosis, cerebral palsy, stroke, or spinal cord injury [3]. BNCI aims to enable users to interact with a computer interface without the use of peripheral nerves and muscles, to restore communication, improve control and possibly enhance social inclusion [4]. Recently BNCI technology has evolved from complex research grade systems to more usable bespoke devices, thereby allowing the recording of Electroencephalographic (EEG) and neuronal activity to become part of the wider self-quantification movement. Swan states: “Analyzing multiple QS (quantified self) data streams in real-time (for example, heart-rate variability, galvanic skin response, temperature, movement, and EEG activity) may likely be required for accurate assessment and intervention regarding biophysical state” [5]. For non-invasive use this has led to a proliferation of cheaper, consumer devices, which can be easily donned and doffed, are more aesthetically pleasing, and use water-based or dry electrodes. Software development kits have become available to the non-specialist, thereby extending domain use into additional lifestyle applications, such as gaming [6] and brain training [7].

Part of the evolution of BNCI has been in the development of hybrid systems which go beyond pure EEG-based paradigms to those that accept multiple inputs from different modalities. Pfurtscheller et al. [8] provide an overview of hybrid Brain-Computer Interface (hBCI) systems, defining concepts and language which has strongly influenced research development in this area. They discuss different ways of combining Brain-Computer Interfaces, with the target of reducing errors, improving available selections, and creating a more usable and robust system. In this paper, we investigate a hBNCI approach, which influences the speed of operation of a graphical interface as measured by Information Transfer Rate (ITR). When an acceptable ITR has been reached, then the collaborative input modalities can be used to ensure more robust operation by reducing errors (paradoxically this may be at the expense of ITR, as damping may occur in the system). However, robustness of operation is a crucial factor for user acceptance, particularly for people with brain dysfunction. In addition, the collection of complementary BNCI and eye tracking data provides the potential for investigation beyond communication and control. Thus the application area for hBNCI can move beyond assistive technology, allowing the exploration of new applications, some of which can be in the medical domain.

2 Background and BNCI Users

Different experimental paradigms can be applied to generate the desired brain electrical activity, known as the electroencephalogram that facilitates the interaction with a chosen computer-based application. Prominent approaches include Event-Related Desynchronisation/Synchronisation (ERD/ERS), Steady State Visually Evoked Potentials (SSVEP), and the P300 oddball paradigm (with acoustic or visual stimulation). Each approach is hindered by its own set of limitations, such as time consuming training and recording, but many inhibiting issues are prevalent in all approaches, such as intra-subject variability, poor signal quality, and limited duration for wearing the technology. These issues have been limiting factors for wider exploitation of BNCI technology in the medical domain. EU FP7 funded projects such as BRAIN, BRAINABLE and Back Home aimed to bring BNCI technology out of the laboratory and into the homes of disabled users. This provided a significant stimulus for addressing communication and control. However, target users involved in the BRAIN study, for example, had cognitive challenges in addition to their physical disability. Furthermore, computer literacy also had an impact on the user acceptance of the technology [9]. In addition to usability issues, poorer BNCI performance was noted in the target user group of brain impaired people, as compared to the healthy control group, and the resulting SSVEP controlled system provided a less than acceptable level of accuracy [10] for the target user group.

3 Hybrid BNCI

There are technical reasons why it could be beneficial to combine different inputs for BNCI. As already highlighted, different modalities have their own merits and drawbacks, which are strongly aligned to the application and user variability. Amiri et al. [11] state: “Compared to other modalities for BCI approaches ….. SSVEP-based BCI system has the advantage of having higher accuracy and higher information transfer rate (ITR). In addition, short/no training time and fewer EEG channels are required.” The BCI component is often used as a switch or selector, for example, see Pfurtscheller et al. [8]: ERD BCI (brain switch) with SSVEP (control of orthosis); ERD combined with SSVEP (joint selection); ERD combined with heart rate (joint selection); Eye gaze (selection) with ERD. In the example of a brain “switch” a control command is only allowed to be activated when a separate BCI control is active. Such a system mitigates the risk of false positives. In terms of “selection” it could be that the two inputs work collaboratively to make a more robust selection. Or, in the example of eye gaze with ERD, the initial selection is made using eye gaze but this decision is activated with ERD [12].

The prospect of combining a neural input with another mechanism such as eye gaze can address under performance issues of BNCI by people with brain dysfunction. Eye tracking-based control was investigated, producing a hybrid architecture, with the potential to overcome restrictions of speed and variability, thus providing a more robust operation [13, 14]. Eye-tracking technology has advanced significantly, producing low cost portable hardware components with open software interfaces mirroring the technical advances of BNCI. Consequently, an hBNCI system has been implemented to facilitate control of a computer interface and virtual domestic smart environment, as illustrated in Fig. 1.

Fig. 1.
figure 1

Hybrid BNCI architecture showing input devices, signal processing options, user interface and actuation components

Full size image

Users have the ability to open and close doors, control the television or indicate needs (need for drinking or eating) or emotions. Combining input modalities with biosignals that have different temporal properties presents a technical challenge in terms of both data fusion and apposite user interface development. However it can offer new opportunities beyond current BNCI.

An experiment was devised to test the robustness of the hybrid approach (albeit on a normal population). Twelve volunteers age 23–57 (8 male, 4 female) interacted with the user interface for three tasks: domotic control; multimedia playback and communication. Interaction was by 4-way choice (right, left, up, select). There were two conditions: eye-tracking only and eye tracking plus BNCI. The Eyetribe Tracker was used to record gaze (latency < 20 ms with an accuracy for 0.5–1 degree, with the subject located approximately 50 cm from the monitor). The Emotiv EPOC provided a BNCI component, using a teeth clench for select. This was chosen as the device comes pre-selected with a number of classified events (appropriate to the static electrode montage of this fixed device) as part of the Expressiv suite. Electronic communication between the eye-tracker/Emotiv headset and user interface is by User Datagram Protocol (UDP) packets, providing a flexible inter-process communication. These are generated/triggered asynchronously (by the participant) and managed by the user interface algorithm, with the slower EEG component acting as a confirmation of the less constrained eye-gaze. The packets are populated in real-time from the respective Eyetribe and Emotiv Application Programming Interfaces (APIs), allowing a responsive and controllable interface. Values for duration, accuracy, efficiency (defined in [15]), and ITR were computed (as defined by Gao et al. [16]). Tables 1 and 2 show the mean and standard deviation for user performance metrics: time, accuracy, efficiency and ITR for eye-tracking only and hybrid respectively. In Tasks 1 and 3 the ITRs are approximately constant but the accuracy and efficiency increase for the hBNCI. In Task 2 the metrics are maintained. Overall accuracy and efficiency are better for the hybrid system.

Table 1. Mean and standard deviation for accuracy, efficiency and information transfer rate for eye-tracker (N = 12)
Full size table
Table 2. Mean and standard deviation for accuracy, efficiency and information transfer rate for hybrid (N = 12)
Full size table

The ITRs of both configurations were greater than that of a previous SSVEP-only study which yielded a mean ITR of 15.23 bpm with a standard deviation of 7.9 bpm and a mean accuracy of 79 % with a standard deviation 14 %. This prior experiment used similar tasks with external stimulation LEDs, to modulate the EEG and assist navigation. However, crucially only 6 out of 23 participants completed all three tasks, which testified to its lack of robustness [17].

Of course this hybrid is based on a low cost commercial headset. It has since been improved by incorporating an SSVEP component or components. The simplest configuration is to use an on-screen SSVEP stimulation as a switch for the eye tracker. However, it is also possible to utilise four stimulation frequencies, allowing for the following navigation options: (i) SSVEP only; and (ii) SSVEP and eye tracking collaborative navigation. The key BNCI components are the quantification of the on-screen navigation and seamless integration with the user interface.

We utilised an intermediary data fusion module to synchronise multimodal interaction and issue a collective command, see Fig. 2. Firstly, the acquired brain signal is computed online for SSVEP signal detection and classification. Nuisance signals and noise are cancelled from the SSVEP response by applying the Minimum Energy Combination method and the best spatial filter for each subject at each frequency is determined automatically by the BCI. The detection of an SSVEP response in the user’s EEG is based on power estimation, which occurs after spatial filtering and a statistical probability method has been applied. This combination enhances separation of the stimulus frequency component in the EEG [14]. When an appropriate SSVEP response is detected, the corresponding command is encapsulated within a UDP packet and forwarded for synchronisation in the data fusion module. At the same time, the eye tracking data is received by the data fusion module as a series of screen-based coordinates. The responsiveness of the eye tracker is dampened to prevent the coordinates buffering and to ensure screen-based coordinates coincide with the trajectory of a user gaze in real-time. If the coordinates do not match the SSVEP response then they are ignored until a matching response is detected. The BCI continuously processes the acquired signal so additional responses can be detected to provide supplementary commands or to rectify false positives. Both input modalities output data concurrently for the entirety of the trial. When both modalities are in agreement a command is classified and encapsulated in another UDP packet, which is transmitted to the graphical interface. At this stage, commands are translated into selections to actuate events in the local environment and provide feedback to the end user, completing the BCI cycle.

Fig. 2.
figure 2

Collaborative processes: fusion and synchronisation of SSVEP-based BNCI and eye tracker information and actuation events in the local environment.

Full size image

4 The Potential of hBNCI for Future Applications

We envisage hBNCI applications along two strands of development. The first will use widely accessible and affordable ‘off the shelf’ BNCI headsets (as demonstrated in the experiment above) with manufacturer supplied software interfaces and development kits. Such kits use dry- or water-based electrodes that can be worn with greater ease. Lifestyle applications include self-quantification for mindfulness or meditation [18], BNCI for HCI in gaming and leisure [19] as enhancement. The second category addresses medical applications using higher quality instrumentation, accessories and robust software with data stored in a standardised format; components that have also benefited from recent technical advances. Medically, BNCI has already been employed for stroke rehabilitation [20–22] and assessing disorders of consciousness [23, 24]. Better quality portable instrumentation can allow for free living assessment and a further example is ambulatory monitoring of EEG for detection of epilepsy or other neurophysiological abnormalities [25, 26] or in sleep studies [27].

For the hBNCI combining modalities (eye gaze for measuring compliance and identification of stimuli, and EEG for measuring engagement) it may be possible to investigate learning for people in classroom scenarios or to investigate conditions such as Dyslexia [28]. A significant contribution can be made in trying to understand the underlying neural cause and triggers associated with mental processing, communication and interaction issues defined as Autistic Spectrum Disorder (ASD). Friedrich et al. have successfully used BCI games for neurofeedback and treatment for children with ASD [29]. A suite of clinical tools were developed within the EU FP7 funded Michelangelo Project [30]. In order to investigate interaction of a child with ASD, a number of elements can be brought together: a task (e.g., an imitation game), engagement with the task (this can be determined from observation, video analysis or directly by measuring eye gaze from the computer and the effect of the task, as measured by physiological signals such as the electrocardiogram (ECG) and EEG). Figure 3 shows the visualisation of synchronized, aggregated data acquired during a task, which permitted therapists and clinicians to better ascertain, or identify, factors contributing to the onset of unwarranted behaviour during the task, thereby leading to personalization of the therapeutic intervention protocol in use.

Fig. 3.
figure 3

Michelangelo project aggregated data visualisation on clinical user interface

Full size image

The clinical tools also permitted further EEG analysis, which comprised off-line, artefact removal (using video playback to identify an appropriate resting state period), followed by event identification, such as eye contact during the task, during which the related EEG signal was processed via clustering techniques in order to identify areas of interest. The clinician is subsequently able to view the results from the analysis, select the appropriate number of synchrostates and visualize the corresponding brain activity for the event [31]. Consequently, such tools, which incorporate EEG as another physiological component, can potentially provide additional insights into both the treatment and understanding of the underlying conditions.

Subsequently, the hBNCI is potentially important for medical applications as it measures complementary biosignals: gaze which can infer attention and task engagement, for example, and brain activity can provide measures of processing of information by the brain. Hence (many) applications for which these components interact can be studied. Controlled psychophysiological studies such as the effect on the EEG of visual semantic content become possible (e.g. the brain’s reaction to food for people with eating disorders [32], visual stimuli for people with addictions such as alcohol and smoking [33]). In addition, it is possible to correlate visual tasks with brain activity for basic research in areas such as monitoring smooth pursuit, saccades, motion onset visual evoked potential and quantification of nystagmus. This may allow further investigation of the vestibule-ocular reflex.

Recent technical advances leading to new lifestyle and novel medical application can extend the reach of BNCI from the specialist laboratory to the neurophysiology clinic and into the living room, thereby engaging a wide user cohort. Abdulkader et al. [34] provide an interesting review of BCI applications and the associated challenges. In reference to the medical domain they classify three streams: prevention, detection and diagnosis, and rehabilitation and restoration. For prevention they cite smoking, alcoholism and motion sickness; for detection and diagnosis they provide examples of tumor detection, brain disorders and sleep disorders; and for rehabilitation they provide examples of brain stoke, disability and psychological disorders.

Brunner et al. [35] provide an overview on how BCI research and European funding in this area has grown over the last ten years and a vision of future BCI. It was expected that passive BCIs would enrich human-computer interaction; BCI tools would be commonly used to support other research domains; and investigations would continue into the possibility of BCI for rehabilitation [36]; and there could be a shift from non-invasive BCIs to invasive BCIs for systems developed to compensate for movement disorders [36].