Abstract
Understanding mental processes in complex human behavior is a key issue in driving, representing a milestone for developing user-centered assistive driving devices. Here, we propose a hybrid method based on electroencephalographic (EEG) and electromyographic (EMG) signatures to distinguish left and right steering in driving scenarios. Twenty-four participants took part in the experiment consisting of recordings of 128-channel EEG and EMG activity from deltoids and forearm extensors in non-ecological and ecological steering tasks. Specifically, we identified the EEG mu rhythm modulation correlates with motor preparation of self-paced steering actions in the non-ecological task, while the concurrent EMG activity of the left (right) deltoids correlates with right (left) steering. Consequently, we exploited the mu rhythm de-synchronization resulting from the non-ecological task to detect the steering side using cross-correlation analysis with the ecological EMG signals. Results returned significant cross-correlation values showing the coupling between the non-ecological EEG feature and the muscular activity collected in ecological driving conditions. Moreover, such cross-correlation patterns discriminate the steering side earlier relative to the single EMG signal. This hybrid system overcomes the limitation of the EEG signals collected in ecological settings such as low reliability, accuracy, and adaptability, thus adding to the EMG the characteristic predictive power of the cerebral data. These results prove how it is possible to complement different physiological signals to control the level of assistance needed by the driver.
Similar content being viewed by others
Avoid common mistakes on your manuscript.
Introduction
Recent advances in sensing and control techniques have made it possible to design cars with a large set of features that can take control of specific aspects of the driving task (e.g., cruise control, autonomous driving) or provide information to the driver to support specific maneuvers (e.g., lane departure). Because driving is a complex behavior involving interrelated motor and cognitive elements such as attention, visuospatial interpretation, visuomotor integration, and decision making (Calhoun et al. 2002; Calhoun and Pearlson 2012), to achieve ever-higher levels of driver support, it is important to investigate and characterize the related brain processes underlying the driver’s actions.
In the years, several attempts were made to characterize the neural correlates of driving in scenarios with different levels of ecology and neural variables, spanning from hemodynamic (Walter et al. 2001; Spiers and Maguire 2007; Mader et al. 2009; Calhoun and Pearlson 2012; Schweizer et al. 2013), to magneto- (Fort et al. 2010; Sakihara et al. 2014) and electrophysiological activity (Schier 2000; Haufe et al. 2011, 2014; Gheorghe et al. 2013; Khaliliardali et al. 2015; Kim et al. 2015; Zhang et al. 2015; Brooks and Kerick 2015; Brooks et al. 2016; Garcia et al. 2017; Vecchiato et al. 2018, 2020). In particular, these latter studies show the possibility of using electroencephalography (EEG) to decode the driver’s cognitive processes in simulated and real car scenarios with the ultimate goal of predicting the upcoming action. Although the success in the classification of salient driving events such as braking (Haufe et al. 2011, 2014; Kim et al. 2014, 2015; Hernández et al. 2018; Wang et al. 2018; Teng et al. 2018; Lin et al. 2018; Nguyen and Chung 2019) and steering (Gheorghe et al. 2013) actions, the level of accuracy is still moderate. Most importantly, the detected neurophysiological features are elicited just around a few milliseconds before the upcoming driving event, making it difficult to implement electronic assisting devices. In fact, despite the noteworthy advancement of recent years, there are still several technological and psychophysiological issues that limit the utilization of the EEG for the real-time identification and monitoring of brain-related activity in driving scenarios. For instance, most EEG sensors remain uncomfortable to place and keep on the head of people who are not used to such recordings, leading to an increase of artifacts and the corruption of the recorded brain signals. Factors such as attention, memory load, and competing cognitive processes (Gonçalves et al. 2006; Käthner et al. 2014; Calhoun and Adali 2016), as well as user’s individual characteristics such as lifestyle, gender, and age (Kasahara et al. 2015) influence brain dynamics producing significant intra- and inter-subject variability (Saha and Baumert 2020; Saha et al. 2021). The low signal-to-noise ratio returned by raw EEG data requires a range of conceptually very different and computationally expensive algorithms to extract significant temporal and frequency EEG features (Müller et al. 2004; Lotte et al. 2007, 2018; Krusienski et al. 2011; Bellotti et al. 2019). Hence, the computing hardware and software must warrant a sufficiently high performance and low latency to preserve the earliness of prediction. For these reasons, it is not always useful to rely on EEG-based predictions alone (Wöhrle et al. 2017). These aspects motivate the need to identify other robust physiological features tackling increased noise due to environmental characteristics and the interaction among neural processes (Lohani et al. 2019). The final aim is to foster the utilization of neurophysiological measurements in experimental settings closer to everyday life activities.
In this sense, surface electromyography (EMG) provides a non-intrusive way of measuring muscle activation and is an appropriate technique when assessing active steering systems (De Luca 1997; Ahlström et al. 2019). It opened new perspectives in ergonomics and provided new tools for analyzing the neuromuscular system in working environments. It is experiencing a growing interest in medical and research applications thanks to the recent availability of novel low-end commercial products increasing the wearability relative to EEG sensors allowing to perform longer recording sessions more comfortably (Gazzoni et al. 2016; Milosevic et al. 2017). Previously, EMG recordings have been used to assess the function of the upper limb muscles during car driving (Jonsson and Jonsson 1975; Liu et al. 2012; Gao et al. 2014). The main findings are that the prime movers are primarily a consequence of steering direction while the stabilizing or fixating muscles are primarily constant, returning that the key muscles correlated to steering are the triceps brachii, the deltoids, pectoralis major, and infraspinatus (Pick and Cole 2006; Liu et al. 2012; Gao et al. 2014). In particular, it is well known that the whole deltoid muscle acts in the abduction of the arm, and there is a synergy between the anterior portion of the muscle and the contralateral posterior portion when moving the steering wheel. More specifically, the anterior portion serves to rotate the steering wheel contralaterally and the posterior portion to rotate it ipsilaterally (Jonsson and Jonsson 1975). EMG can be a reliable method for action prediction because it has limited sensitivity to environmental disturbance (Bi et al. 2019). Compared to EEG, surface EMG can be easily acquired and processed and provide useful information on the movement that the person is executing. Despite the advantages mentioned above, EEG and EMG exhibit different temporal properties concerning a movement onset: changes in the EMG can only be observed close to the actual movement onset, while it is possible to detect EEG motor biomarkers earlier (Bai et al. 2011; Wöhrle et al. 2017; Trigili et al. 2019).
It has been demonstrated that voluntary movements generate changes in the alpha and beta ranges of the EEG spectrum (Pfurtscheller and Lopes da Silva 1999) and that such rhythms are in turn related to modulations of the EMG activity (Hari and Salenius 1999). Recent works have focused on the synchronization between rhythmical activity in the motor cortex and muscular activity employing cortico-muscular coherence, which is usually observed during periods of muscular contraction and has been reported in several studies involving both EEG and MEG (Conway et al. 1995; Boonstra et al. 2009; Cheyne 2013; Rizzo et al. 2020). Current approaches to cortico-muscular coordination focus on associations and synchronous activation between individual brain rhythms at specific cortical areas (e.g., motor cortex, hippocampus), and peripheral muscle activity during specific movement tasks or exercises in ecological conditions (e.g., walking and running) (van Wijk et al. 2012; Cui et al. 2017; Rendeiro and Rhodes 2018; Li et al. 2019a, 2020; Fauvet et al. 2019). In such scenarios, the neuromuscular signals are noisier than those collected in standard laboratory conditions due to uncontrolled environmental settings, and there is also the issue of addressing the simultaneous presence of concurrent cerebral processes. Finally, on the user side, there is the need to create comfortable experimental settings to not pollute the neuromuscular signals with undesired components due to the experienced fatigue of wearing biomedical sensors on the head and several parts of the body. To overcome issues with both EEG- and EMG-based control methods, a combination of both systems, building on each signal’s advantages and diminishing the limitations of each, might be a promising strategy (Lalitharatne et al. 2013). It would then be optimal to identify the cerebral features of interest in standard laboratory settings and then look for neuromuscular invariants in more ecological settings with higher recording complexity. In this way, offline analysis performed on data collected in standard settings would inform the online pipeline to implement to extract neuromuscular features of interest.
Following this reasoning, we propose a hybrid method to distinguish left from right steering in a driving simulator based on EEG and EMG signals collected during steering actions. While using EEG and EMG signals in steering decoding would certainly add predictive power relative to unimodal recordings, such a strategy is hurdled by the complexity of combining the two acquisitions during real-life driving. However, a potential solution could be to first identify in a preliminary recording the steering-related EEG features and then use such information during real driving to increase the predictive power of the steering action based solely on the EMG acquisition.
We used two experimental scenarios to demonstrate this possibility: first, participants were required to perform stand-alone steering wheel movements; successively, we used a driving simulator that represents a step forward towards recording the continuous EEG signals in the real car in natural traffic conditions. This difference between the experimental conditions led us to adopt the terms “non-ecological” and “ecological” steering task. We hypothesized that the electrophysiological correlates of the non-ecological steering could assist the steering action detection in the ecological condition.
Starting from these premises, we first investigated brain and muscular activity underlying steering behavior during the non-ecological steering task. This procedure allowed us to identify the EEG correlates of steering without confounding effects. Then, we correlated such features with the EMG activity collected during a session of driving simulation to extend the validity of the non-ecological cerebral signatures to a more ecological steering task. This double task approach enabled to (i) take advantage of multimodal recordings exploiting the information carried by both neural and muscular data, and (ii) solve ergonomic issues related to the simultaneous acquisition of different signals.
Therefore, we aimed to characterize the EEG–EMG coupling associated with the natural and self-initiated execution of steering actions while driving. To this end, we took advantage of the independent component analysis (ICA), which separates mixed EEG signals into maximally independent activities, each characterized by a precise scalp topography and a corresponding generator pattern, typically modeled as patches of cortical pyramidal cells (Delorme et al. 2012). This strategy also allowed us to avoid field spread caused by the large distance between sensors and neural sources and by the spatial blurring effect of the skull on the scalp’s potential distribution of EEG signals (Schoffelen and Gross 2009). Thus, we expect to identify reliable, independent components whose topography indicates the involvement of motor circuits, whose reactivity precedes the steering action, and is correlated with the muscular activity of the deltoids. Should the EEG–EMG coupling be modulated across the two types of action participants perform (i.e., left and right steering), this would prove that such a hybrid method can be used to discriminate and predict motor intentions associated with natural driving behavior.
Material and methods
Participants
Twenty-four participants (6 females, M: 22.8 ± SD: 2.0 years of age) participated in this experiment. They verbally declared that they were right-handed and had a normal or corrected vision, no history of neurologic or psychiatric disorders, and no daily medications. They all have a driving license obtained from the Italian state authority for motor vehicles. None of them had driving experience on professional racetracks. They gave written informed consent to participate in the study consisting of two consecutive sessions comprising a non-ecological steering task and a following ecological steering task. Approval for the study was obtained by the local Ethical Committee (comitato etico Unico per la provincia di Parma).
Experimental scenarios
Non-ecological steering task
Participants seated in front of a computer screen at a distance of 1 m and were instructed to turn at a quiet pace the steering wheel (Logitech G25) on the right or the left according to a traffic sign randomly presented on display after 1 s from an attentional cue (i.e., a white cross). Participants who performed the steering within 2 s from the traffic sign received an error message and repeated the trial. According to this rule, 88 correct trials have been collected for each participant, equally distributed between left and right steering actions. Synchronization between the visual stimulation, participant’s behavior, and EEG data was realized using timestamps sent by a photodiode placed in front of the screen to capture the task events through white labels onto a black background. Figure 1 shows the experimental setup (panel A) and the time-course of the task (panel B).
Ecological steering task
Participants were seated in a driving simulator composed of the RSeat RS1 Assetto Corsa Special Edition (seat) and Thrustmaster T500 RS (steering wheel and pedals). A video game (Assetto Corsa—Kunos Simulazioni) was displayed on a Samsung 40” 5300 class LED TV positioned 1 m from the participant’s seat (vertical field view = 27.4°, horizontal field view = 46.8°). The car used for the experiment was an Alfa Romeo Mito with an automatic transmission. First, participants were asked to get familiar with the environment and the setup by driving for a lap on the Monte Erice track (downloadable at http://assettocorsa.club/mods/tracks/monte-erice.html). Afterward, they performed a single lap on the Coste Loop track (part of the Assetto Corsa software), maintaining the right lane with no particular constraints. They were asked to drive naturally, as they would do in their own car. This circuit simulates a part of the Garda Lake coastal road. No other vehicle was present on both tracks. During the experiment, we collected data related to the steering wheel angle. The synchronization among all the recording devices was implemented with the Lab Streaming Layer (LSL) as described in our previous work (Vecchiato et al. 2018). Figure 1 reports the experimental setup (panel C) and the Coste Loop track (panel D).
Behavioral data collection and analysis
The steering wheel signals were collected during the non-ecological and ecological tasks and segmented in trials [− 2000, 2000] ms around the steering onset. In order to identify the steering onset in the non-ecological task, we consider the first part of the trial returning wheel angles above the threshold of ± 2°. Of this segment, the time bin corresponding to the point of maximum distance computed from the steering curve and the segment joining the first and last trial sample was identified as the steering onset (see Supplementary Material for a schematic representation of the procedure). When uncertainty in the movement onset arose, we discarded the trial (around 3.5%). This procedure also served for epoching the EEG and EMG signals collected during the non-ecological task into [− 2500, 2500] ms trials around the steering onset.
Instead, in the ecological task, the steering onset was determined by quantizing the steering wheel angle signal. The quantization was realized partitioning the wheel signal with a step of 0.125 between its minimum and maximum. This step was developed to verify that the area-under-the-curve for original and quantized signals differed less than 5%, and their correlation coefficient was above 99%. The quantized signal was binarized setting at 1 all values greater than 0 (in absolute value) and clustered. The steering onset was determined as the time bin corresponding to the transition between zeros and ones. With such procedure, we identified 94.58 ± 31.12 right and 79.08 ± 23.85 left steering trials.
Moreover, the steering wheel signal collected during the ecological task was also segmented during the non-steering intervals in [− 2000, 2000] ms overlapping trials of 1 s, thus identifying 163.71 ± 103.55 trials.
EEG and EMG data recording and pre-processing
Continuous EEG was recorded in the non-ecological task using the 128-channel Geodesic EEG System (Electrical Geodesics, Inc., Eugene, OR, USA) and the HydroCel Geodesic Sensor Net. Consistent positioning was achieved by aligning the Sensor Net with skull landmarks (nasion, vertex, and pre-auricular points). Using high-input impedance amplifiers (Net Amps300), low-noise EEG data was obtained with sensor-skin impedances maintained below 50 kΩ. The signal was digitized at a sampling rate of 500 Hz (0.01 Hz high-pass filter) and recorded with a vertex reference, the impedance of which was kept below 10 kΩ. Impedances were checked and adapted at the beginning of the ecological steering task. EEG data were exported in raw format using NetStation software (Electrical Geodesics, Inc., Eugene, OR, USA) and then imported into MATLAB to perform the following analysis with EEGLAB v14.1.2 (Delorme and Makeig 2004). The pre-processing comprised line noise removal, bad channels interpolation (1.1 ± 1.1), and common average reference. EEG data were segmented into epochs [− 1.5, 11.5] s around the presentation of the steering traffic sign to consider both the pre-stimulus activity and the one associated with self-paced movements dynamic extending beyond 10 s. Artifacts were rejected by applying a semi-automatic procedure to detect abnormal trends and spectra. On average, we discarded 11.4 ± 7 trials. Clean EEG datasets comprised 37.6 ± 3.7 left and 37.7 ± 3.3 right steering trials.
Continuous EMG data in both non-ecological and ecological steering tasks were acquired using the Neuroelectrics Enobio. EMG signals were sampled at 500 Hz from the left and right deltoids, left and right forearm extensor digitorum as among the main muscles involved in steering actions (Pick and Cole 2006; Lohani et al. 2019) and later imported and processed in MATLAB environment (R2018b, The Mathworks, Natick, MA). EMG data of the non-ecological task were segmented into epochs [− 1.5, 11.5] s around the presentation of the steering traffic sign (as done for the EEG data). In contrast, EMG data of the ecological task were segmented into two datasets: (i) epochs [− 2.5, 2.5] s around the onset of the steering action, as well as (ii) 1 s overlapping epochs of [− 2.5, 2.5] during non-steering intervals. Line noise of the first 5 harmonics of 50 Hz was suppressed using a spectrum estimation technique (Mewett et al. 2001).
To summarize, we have two different segmentations related to the non-ecological ([− 1.5, 11.5] s around the steering sign presentation) and ecological conditions ([− 2.5, 2.5] s around the steering onset). We used different time scales in the results to account for analyses purposes.
EEG data analysis and results related to the ecological steering task were provided as Supplementary Material.
EMG and EEG data analysis
EEG independent component and clustering analysis
We performed an ICA to identify and separate neurophysiological brain activities from other noise sources. On average, we identified 5.9 (± 2.1) independent components (ICs) per subject for a total of 147 EEG ICs. These brain-related IC were identified through equivalent dipole source localization (DIPFIT2) and the utilization of the SASICA EEGLAB plugin (Chaumon et al. 2015). Cluster analysis was performed to group components according to their scalp topographies via the K-means algorithm (Lloyd 1982). The dimension of the cluster space was set to 109, which is the minimum number of good EEG channels remaining across participants after pre-processing. The correlation Squared Euclidean distance is the metric used for minimization measure was chosen to compute cluster centroids using the following formula:
where x is the vector identifying the single-subject scalp map and c is a centroid. The number of clusters was chosen by a bootstrap-based method, leading to the computation of a stability index (Ben-Hur et al. 2002; Salvador and Chan 2004) and identifying 5 clusters. The rationale of using this data-driven analysis was to identify those brain components related to steering actions. Hence, the cluster analysis allowed grouping those EEG ICs related to different cerebral processes, while the statistical comparisons between left and right conditions allowed identifying the cluster reacting to steering actions. Details related to the pre-processing pipeline, ICA, and clustering can be found in our previous work (Vecchiato et al. 2018).
Time–frequency analysis
For each EMG signal and EEG IC within the identified clusters, we computed the event-related spectral perturbation (ERSP) as time–frequency decomposition using Morlet wavelets (Makeig et al. 2004). We used frequencies that increased from 2 to 200 Hz in 198 linearly spaced steps for the EMG data, with the number of wavelet cycles increasing from 3 to 60 in linear steps. We used frequencies that increased from 2 to 40 Hz in 38 linearly spaced steps for the EEG data, with the number of wavelet cycles increasing from 3 to 12 in linear steps. Then, dB conversion was performed with a single-trial baseline normalization using the [− 0.5, − 0.1] s time window before the steering sign presentation for the non-ecological steering task and the interval of [− 2, 2] s for the ecological paradigm.
This procedure allowed us to investigate EEG and EMG correlates of steering actions in both time and frequency domains.
Cross-correlation analysis
To identify the possible time lag between the EEG signals recorded in the non-ecological task and the EMG signal recorded in the ecological task, we computed the bi-dimensional cross-correlation between the corresponding time–frequency panels using the xcorr2 function provided by Matlab, which is based on a bi-dimensional convolution between the two input matrices. In particular, for each subject and steering condition in the non-ecological task, we extracted the EEG mask identified within the time–frequency panel around [8, 20] Hz and [− 1.5, − 1] s before the steering onset (e.g., left non-ecological steering), whose dimension is 13 × 50 (frequency x time). Then, we performed the bi-dimensional cross-correlation trial-by-trial for each subject between such EEG mask and the EMG time–frequency panel within the time range [− 1, 2] s (e.g., left ecological steering), whose dimension is 198 × 300. The output of this calculation is a matrix of dimension 210 × 349, from which we discarded the first 12 rows related to non-significant frequencies. Hence, the final cross-correlation matrix used for the following statistical analysis had dimension 198 × 349.
The specificity of the resulting cross-correlation patterns for the steering actions was addressed by computing the same analysis using the EMG trials related to non-steering intervals as defined in the previous section. In particular, EMG non-steering trials were randomly assigned to left and right pseudo-steering conditions with a half split technique: for each of 300 iterations, non-steering trials were divided into two sets, only one of these subsets was used to create the left and right pseudo-steering conditions. Then, for each iteration, we computed the cross-correlation values with the EEG mask resulting from the analysis of the non-ecological dataset. Finally, we also tested the specificity of the cross-correlation patterns for the directionality of the steering by adopting a shuffling procedure. For each of 300 iterations, we created the pseudo-left and pseudo-right conditions by randomly assigning EMG trials from the original left and right steering datasets to the two pseudo-steering conditions and then computing the cross-correlation values.
This procedure allowed us to estimate the contribution of the EEG mask in predicting the EMG activity of the single muscles (i.e., deltoids and forearms extensors) and experimental conditions (i.e., left vs. right steering, steering vs. non-steering, steering vs. shuffled-steering).
Statistical analysis
To discriminate the EEG and EMG activity in both non-ecological and ecological tasks, the corresponding time–frequency panels were compared using dependent sample t-statistics and non-parametric permutation testing, corrected for multiple comparisons by weighted cluster mass correction with randomization of 1000 and a statistical threshold of 0.05 (Hayasaka and Nichols 2004; Maris and Oostenveld 2007). The statistical comparison between left and right EEG IC ERPSs in the non-ecological task allows identifying and extracting the time–frequency EEG features associated with steering actions and is used for the following cross-correlation analysis. The significance of the t-statistics computed to test the specificity for steering and directionality was assessed by comparing the observed statistics to the statistical properties of the null-hypothesis distribution. Hence, the observed test statistic values were converted into Z scores, and then the corresponding p-values were computed and reported (Cohen 2014).
This procedure allowed us to assess the significance of the predictive power of the EEG mask for steering actions and directions.
Results
Non-ecological steering task
For each cluster of EEG ICs, we performed the non-parametric permutation test to compare the ERSP between the two steering conditions (i.e., left vs. right) in the time window of [− 2, 2] s relative to the onset of the event. The analysis revealed a statistically significant difference onset only for cluster 3, highlighting a de-synchronization of the mu rhythm around 1.5 s before the left steering. This cluster is populated by 24 ICs belonging to 16 subjects (illustrated in the Supplementary Material). Figure 2 shows the scalp topography of such a significant cluster along with the modulation of the related time–frequency activity during left (panel A) and right (panel B) non-ecological steering with the corresponding t-statistics (panel C). Centroid maps with the corresponding time–frequency panels of the non-significant EEG IC clusters are illustrated in the Supplementary Material.
Figure 2 also shows the ERSP of the EMG activity for the left (panels D, E) and right deltoids (panels G, H) during the left (panels D, G) and right (panels E, H) steering. Specifically, when comparing these signals between left and right steering, we observed the significant broadband increase of the muscular activation of the left deltoid during the right steering (panel F), as well as the symmetrical broadband increase of activity of the right deltoid during the left steering (panel I). Analogously, when comparing these signals between left and right deltoids, we observed the significant broadband increase of the muscular activation of the right deltoid during the left steering (panel J), as well as the symmetrical broadband increase of activity of the left deltoid during the right steering (panel K). The statistical comparisons of the muscular activation of the two forearm extensors did not return significant differences in the EMG broadband as the two deltoids (see Supplementary Material, Figure S3); thus they are not considered in the following analysis.
Variations of the wheel angle for the left and right steering were represented with the white signals within each corresponding panel of Fig. 2.
Ecological steering task
Figure 3 shows the results related to the analysis of the EMG signals during the ecological task performed at the driving simulator. In particular, the upper box presents the ERSP computed for the EMG signals of the left and right deltoids during left and right steering actions. The different panels of this figure are arranged as Fig. 2 to show that the asymmetrical activation of the two deltoids during steering observed in the non-ecological task is replicated in the ecological condition. Specifically, we highlight the significant broadband increases of the muscular activation of the left deltoid during the right steering (panels A–C) and the symmetrical broadband increase of activity of the right deltoid during the left steering (panels D–F). Analogously, when comparing these signals between left and right deltoids, we observed the significant broadband increase of the muscular activation of the right deltoid during the left steering (panel G), as well as the symmetrical broadband increase of activity of the left deltoid during the right steering (panel H).
The lower box of Fig. 3 shows the results related to the cross-correlation analysis performed between the EEG mask gathered during the non-ecological task and the EMG ERSP panel corresponding to the ecological scenario in the same steering condition and for each of the two deltoids (i.e., non-eco EEG in left steering cross-correlated with the eco EMG in left steering). Here we can observe higher cross-correlation values associated with the left deltoid during left steering (panel I) and for the right deltoid during the right steering (panel M) when compared with the right (panel J) and left (panel L) steering, respectively. This pattern is statistically demonstrated with the corresponding non-parametric analysis (panel K, N). Moreover, such differences between cross-correlation values are also observed during left (panel O) and right (panel P) steering when comparing the left and right deltoid. Strikingly, these statistics show that significant activations related to the cross-correlation analysis are detected earlier (magenta contour in panel Q) relative to the activations of only EMG ERSP (blue contour in panel Q), for the condition right deltoid, left versus right steering.
Figure 4 shows the results of the cross-correlation analysis performed between the same EEG mask gathered during the non-ecological task, and the EMG ERSP panels corresponding to the ecological scenario in the non-steering (upper box) and shuffled-steering (lower box) conditions. Here we can observe low cross-correlation values associated with all conditions reported on the same scale of Fig. 3 (lower box). Finally, the corresponding statistics returned no significant difference for any condition.
Variations of the wheel angle for the left and right steering were represented with the white (Fig. 3) and black (Fig. 4) signals within each corresponding panel.
The analysis of the EEG collected during the ecological steering task did not return any significant cluster related to cortical components reacting to steering actions. Thus, we report the corresponding ERPSs and statistical comparisons in the Supplementary Material.
Discussion
In the present study, we report that the modulation of the EEG mu rhythm observed during the motor preparation of non-ecological steering predicts the muscular activity of deltoids, thus anticipating subject steering behavior. The reactivity of such rhythm measured across sensorimotor areas during the non-ecological steering preparation anticipates the corresponding action. This result paves the way for using such a cerebral feature to discriminate steering actions in the ecological task. We report the increase of EMG activity of the deltoid anticipating the contralateral steering in non-ecological and ecological steering tasks. These results show an asymmetric muscular activity of the deltoids beginning before the action onset remaining steady during the steering execution, i.e., the coordinated increase of power of the right deltoid and the corresponding decrease of power of the left deltoid is associated with the steering action on the left side. Such findings show that monitoring the two deltoids’ muscular activity makes it possible to discriminate the steering side before the action onset while driving in non-ecological and ecological scenarios. Strikingly, the identified non-ecological EEG feature correlates with the ecological EMG activity of the deltoids, providing an improvement of the discrimination power of the steering side during driving simulation. The comparison between the results related to the EMG analysis and those concerning the EEG–EMG cross-correlation returned larger relative timing in favor of the latter related to the succession of steering events. In fact, on one side, there is the issue of distinguishing the anticipatory components due to EEG predicting power from those which are merely due to computational accounts, e.g., windowing implied in the cross-correlation calculation. However, we observe a clear anticipatory pattern returned by cross-correlation values related to the succession between left and right steering events. This change of cross-correlation values is detectable in advance relative to the single EMG signal.
The timing of the EEG response identified in this study is compatible with patterns of event-related de-synchronization (ERD) which are reported to be predictive of the upcoming action between 1.5 and 1 s before the movement onset (Neuper et al. 2006). ERD reflects non-phase-locked EEG changes of oscillatory activity within the alpha and beta frequency due to sensorimotor events (Pfurtscheller and Lopes da Silva 1999; Savić et al. 2020). The EEG component identified during the non-ecological task arises from the motor areas and may reflect the preparation of steering actions. The scalp topography that we reported is reminiscent of the mu rhythm observed in motor and premotor regions (Gastaut et al. 1952; Pfurtscheller et al. 1997). Such electrical activity produces somatotopically organized de-synchronization during execution, observation, and imagination of actions (Pineda 2005; Pfurtscheller et al. 2006; Arnstein et al. 2011; Avanzini et al. 2012). These regions perform several functions other than body movements control, such as sensory-motor transformation, action understanding, decision-making regarding execution and initiation of action, preparation, and planning of complex movements (Roland 1984; Rizzolatti and Luppino 2001). Recent literature reports ERD in contralateral sensorimotor cortices during movement preparation of visually cued movements (Li et al. 2018; Little et al. 2019). Therefore, we argue that the identified EEG mu rhythm modulations regulate the motor preparation of the right and left deltoids for steering actions. In particular, we report that the mu de-synchronization over left sensorimotor regions is related to the right deltoid’s activity during left steering. This would show that left steering involves larger neural computation than right steering, despite the absence of significant difference in the steering wheel angle, as already reported in a previous study (Oka et al. 2015). Another study also reports the de-synchronization of the alpha rhythm across sensorimotor regions related to relative steering angle compensation (Brooks and Kerick 2015), thus showing the relation between such electroencephalographic feature and steering response as already observed in more simple tasks (Pfurtscheller and Neuper 1994; Stancák and Pfurtscheller 1996).
Analyzing the EMG in the time–frequency domain, we observe that the maximum muscle activity changes significantly due to different steering wheel angles and turning directions in non-ecological and ecological scenarios. It is known that the anterior and middle portions of the deltoid muscle work intermittently during car driving and that their functioning regulates the contralateral rotation of the steering wheel, being activated for a duration of around 50% since the initial stage of the action (Jonsson and Jonsson 1975; Pick and Cole 2006; Liu et al. 2012; Gao et al. 2014). Specifically, muscle activity is relatively small when the steering wheel is near its center, but it increases rapidly as the wheel starts to turn (Gao et al. 2014). Although EMG was successfully used to identify the muscles involved in generating and predicting torque at the steering wheel (Pick and Cole 2006), EMG alone has low power in predicting the steering as we report to be limited to a few hundred milliseconds. Thus, we exploited the EEG scalp feature to improve steering detection by computing the cross-correlation between the mu rhythm de-synchronization retrieved during the non-ecological task and the EMG activity elicited in the ecological scenario. This procedure allowed us to enlarge the time window in which it is possible to discriminate the steering action. We showed that this EEG–EMG coupling is specific for steering actions and their directionality, returning better results relative to single EMG signals. Similar findings were recently reported concerning the use of a hybrid human–machine interface for gait decoding (Tortora et al. 2020), motor rehabilitation of stroke patients (Sarasola-Sanz et al. 2017), movement detection of hand-paralyzed patients (Lóopez-Larraz et al. 2018). All these studies suggested an increase in the classification accuracy and the number of commands for human–machine-interfaces (Hong and Khan 2017).
Here we performed two consecutive recording sessions to demonstrate the feasibility of exploiting EEG correlations of steering behaviour collected in a standard highly controlled environment to detect steering action in a more ecological setting such as a driving simulator. In the first non-ecological recording session, we extracted ERD as a neural correlate of the EMG activity of the deltoids predicting the upcoming steering behavior. In the following ecological recording session performed at the driving simulator, we collected the EMG activity of the deltoids. We showed that the ERD resulting from the previous session is indeed informative concerning the steering action in this more naturalistic scenario. As expected, a large variability characterized the EEG collected during the ecological steering task, and the statistical analysis did not return any significant result related to the cerebral steering component. Indeed, steering actions were reliably discriminated through the analysis of the cortical correlates associated with the non-ecological task. This evidence suggests the usefulness of exploiting cortical correlates of motor preparation of non-ecological actions to predict the same action in ecological conditions. The non-significant results of the EEG collected during the ecological steering task indeed highlight the usefulness of exploiting cortical correlates associated with the non-ecological task for steering detection in a more natural condition. From a methodological perspective, we used the ERSP to investigate EMG correlates of steering for directly assessing the coupling of cerebral and muscular activations. EMG features in the time domain, such as Root Mean Square and Mean Absolute Value, could also be useful for future analyses implementing online classifiers.
Several recent studies on driving addressed the issue of steering classification using different EEG features. In particular, it was assessed the possibility to decode self-generated actions detecting whether the driver would perform a lane change shortly in a simulated highway (Gheorghe et al. 2013). Authors report slow negative EEG deflections across central areas consistent with the movement-related potentials 500 ms before the lane change, yielding a classification accuracy of 79% with an average detection time of 613 ms before the actual steering action. In another series of studies performed with both driving simulators and real cars (Zhang et al. 2015), error-related brain potentials were analyzed to investigate the possibility of using an external device to be adapted to the driver’s goal, i.e., assisting in the upcoming steering action. This strategy was enacted by showing the drivers a visual stimulus indicating their inference about the next turning direction when approaching an intersection. Authors report differences in the EEG response over fronto-central areas when the directional stimulus does not match the driver’s intention. Statistical differences between error and correct conditions were observed between 200 and 600 ms after feedback, yielding a mean accuracy of the event-related decoding of 0.68, which indicates the possibility of extracting meaningful information about the driver’s need for assistance.
Several studies demonstrated that the combination of EEG and EMG can improve the reliability of movement prediction based on a single modality (Vecchiato 2021; Di Liberto et al. 2021). For example, the prediction of movement onset based on EEG analysis can be improved by designing hybrid systems monitoring at the same time additional peripheral signals depending on the context requirements. EEG and EMG signals can reliably predict movements before the action onset, showing that both can potentially control an electronic device (Kirchner et al. 2014). Different measures could be combined for different movement stages, e.g., movement planning, the start of the movement, and movement execution (Novak et al. 2013). Other investigations were conducted to predict voluntary movements before their occurrence (Bai et al. 2011) and vehicle steering (Gomez-Gil et al. 2011) using hybrid EEG–EMG BCIs. The hybrid strategy was initially introduced in the Brain Computer Interfaces (BCIs) to exploit the advantages of different physiological signals and computational approaches to achieve specific goals better than a conventional EEG-based system (Pfurtscheller et al. 2010; Li et al. 2019b). A hybrid system might predict user intention with higher accuracy, thus improving the whole system’s performance or reducing the rate of false positives (Usakli et al. 2009, 2010; Ma et al. 2015; Hong and Khan 2017). Today such a methodology is used to boost motor rehabilitation of stroke patients (Sarasola-Sanz et al. 2017), for the online movement prediction (Kirchner et al. 2014; Wöhrle et al. 2017), gait decoding (Tortora et al. 2020), and its application is investigated in other several domains of bio-robotics (Lalitharatne et al. 2013).
The existence of preparatory electrophysiological activity elicited before the onset of steering action allows us to infer the upcoming driving actions in advance. The reported EEG–EMG coupling is a proof of concept for utilizing hybrid systems for the detection and online prediction of driving actions, exemplifying how it might be possible to complement information from behavioral, physiological, and external sources to control the level of assistance needed by the driver in that context (Chavarriaga et al. 2018). The predictive power of the EEG–EMG coupling demonstrated in a car simulator could be further investigated in larger sets of actions to extend the validity of this neurophysiological mechanism beyond driving.
Data availability
The data that support the findings of this study are available from CAMLIN Limited but restrictions apply to the availability of these data, which were used under licence for the current study, and so are not publicly available. Data are however available from the authors upon reasonable request and with permission of CAMLIN Limited.
Code availability
Custom code is available upon reasonable request.
References
Ahlström C, Gink Lövgren M, Nilsson M et al (2019) The effect of an active steering system on city bus drivers’ muscle activity. Int J Occup Saf Ergon 25:377–385. https://doi.org/10.1080/10803548.2018.1445465
Arnstein D, Cui F, Keysers C et al (2011) μ-Suppression during action observation and execution correlates with BOLD in Dorsal premotor, inferior parietal, and SI cortices. J Neurosci 31:14243–14249. https://doi.org/10.1523/JNEUROSCI.0963-11.2011
Avanzini P, Fabbri-Destro M, Volta RD et al (2012) The dynamics of sensorimotor cortical oscillations during the observation of hand movements: an EEG study. PLoS ONE 7:e37534. https://doi.org/10.1371/journal.pone.0037534
Bai O, Rathi V, Lin P et al (2011) Prediction of human voluntary movement before it occurs. Clin Neurophysiol 122:364–372. https://doi.org/10.1016/j.clinph.2010.07.010
Bellotti A, Antopolskiy S, Marchenkova A et al (2019) Brain-based control of car infotainment. 2019 IEEE international conference on systems, man and cybernetics (SMC). IEEE, Bari, pp 2166–2173
Ben-Hur A, Elisseeff A, Guyon I (2002) A stability based method for discovering structure in clustered data. Pac Symp Biocomput 6–17
Bi L, Feleke AG, Guan C (2019) A review on EMG-based motor intention prediction of continuous human upper limb motion for human-robot collaboration. Biomed Signal Process Control 51:113–127. https://doi.org/10.1016/j.bspc.2019.02.011
Boonstra TW, van Wijk BCM, Praamstra P, Daffertshofer A (2009) Corticomuscular and bilateral EMG coherence reflect distinct aspects of neural synchronization. Neurosci Lett 463:17–21. https://doi.org/10.1016/j.neulet.2009.07.043
Brooks J, Kerick S (2015) Event-related alpha perturbations related to the scaling of steering wheel corrections. Physiol Behav 149:287–293. https://doi.org/10.1016/j.physbeh.2015.05.026
Brooks JR, Garcia JO, Kerick SE, Vettel JM (2016) Differential functionality of right and left parietal activity in controlling a motor vehicle. Front Syst Neurosci. https://doi.org/10.3389/fnsys.2016.00106
Calhoun VD, Adali T (2016) Time-Varying Brain Connectivity in fMRI Data: whole-brain data-driven approaches for capturing and characterizing dynamic states. IEEE Signal Process Mag 33:52–66. https://doi.org/10.1109/MSP.2015.2478915
Calhoun VD, Pearlson GD (2012) A selective review of simulated driving studies: combining naturalistic and hybrid paradigms, analysis approaches, and future directions. Neuroimage 59:25–35. https://doi.org/10.1016/j.neuroimage.2011.06.037
Calhoun VD, Pekar JJ, McGinty VB et al (2002) Different activation dynamics in multiple neural systems during simulated driving. Hum Brain Mapp 17:141–142. https://doi.org/10.1002/hbm.10066
Chaumon M, Bishop DV, Busch NA (2015) A practical guide to the selection of independent components of the electroencephalogram for artifact correction. J Neurosci Methods 250:47–63. https://doi.org/10.1016/j.jneumeth.2015.02.025
Chavarriaga R, Ušćumlić M, Zhang H et al (2018) Decoding neural correlates of cognitive states to enhance driving experience. IEEE Trans Emerg Top Comput Intell 2:288–297. https://doi.org/10.1109/TETCI.2018.2848289
Cheyne DO (2013) MEG studies of sensorimotor rhythms: a review. Exp Neurol 245:27–39. https://doi.org/10.1016/j.expneurol.2012.08.030
Cohen MX (2014) Analyzing neural time series data | The MIT Press. Available at https://mitpress.mit.edu/books/analyzing-neural-time-series-data. Accessed on 8 Jun 2021
Conway BA, Halliday DM, Farmer SF et al (1995) Synchronization between motor cortex and spinal motoneuronal pool during the performance of a maintained motor task in man. J Physiol 489(Pt 3):917–924. https://doi.org/10.1113/jphysiol.1995.sp021104
Cui C, Bian G-B, Hou Z-G et al (2017) A multimodal framework based on integration of cortical and muscular activities for decoding human intentions about lower limb motions. IEEE Trans Biomed Circuits Syst 11:889–899. https://doi.org/10.1109/TBCAS.2017.2699189
De Luca CJ (1997) The use of surface electromyography in biomechanics. J Appl Biomech 13:135–163. https://doi.org/10.1123/jab.13.2.135
Delorme A, Makeig S (2004) EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J Neurosci Methods 134:9–21. https://doi.org/10.1016/j.jneumeth.2003.10.009
Delorme A, Palmer J, Onton J et al (2012) Independent EEG sources are dipolar. PLoS ONE 7:e30135. https://doi.org/10.1371/journal.pone.0030135
Di Liberto GM, Barsotti M, Vecchiato G et al (2021) Robust anticipation of continuous steering actions from electroencephalographic data during simulated driving. Sci Rep 11:23383. https://doi.org/10.1038/s41598-021-02750-w
Fauvet M, Cremoux S, Chalard A et al (2019) A novel method to generalize time-frequency coherence analysis between EEG or EMG signals during repetitive trials with high intra-subject variability in duration. 2019 9th International IEEE/EMBS Conference on Neural Engineering (NER). IEEE, San Francisco, pp 437–440
Fort A, Martin R, Jacquet-Andrieu A et al (2010) Attentional demand and processing of relevant visual information during simulated driving: a MEG study. Brain Res 1363:117–127. https://doi.org/10.1016/j.brainres.2010.09.094
Gao Z, Fan D, Wang D et al (2014) Muscle activity and co-contraction of musculoskeletal model during steering maneuver. Bio-Med Mater Eng 24:2697–2706. https://doi.org/10.3233/BME-141087
Garcia JO, Brooks J, Kerick S et al (2017) Estimating direction in brain-behavior interactions: proactive and reactive brain states in driving. Neuroimage 150:239–249. https://doi.org/10.1016/j.neuroimage.2017.02.057
Gastaut H, Terzian H, Gastaut Y (1952) Etude d’une activité électroencéphalographique mécconue: “Le rythme rolandique en arceau.” Mars Med 89:296–310
Gazzoni M, Afsharipour B, Merletti R (2016) Surface EMG in ergonomics and occupational medicine. Surface electromyography: physiology, engineering, and applications. Wiley, Hoboken, pp 361–391
Gheorghe L, Chavarriaga R, Millán JDR (2013) Steering timing prediction in a driving simulator task. 2013 35th annual international conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE, Osaka, pp 6913–6916
Gomez-Gil J, San-Jose-Gonzalez I, Nicolas-Alonso LF, Alonso-Garcia S (2011) Steering a tractor by means of an EMG-based human-machine interface. Sensors 11:7110–7126. https://doi.org/10.3390/s110707110
Gonçalves SI, de Munck JC, Pouwels PJW et al (2006) Correlating the alpha rhythm to BOLD using simultaneous EEG/fMRI: inter-subject variability. Neuroimage 30:203–213. https://doi.org/10.1016/j.neuroimage.2005.09.062
Hari R, Salenius S (1999) Rhythmical corticomotor communication. NeuroReport 10:R1-10
Haufe S, Treder MS, Gugler MF et al (2011) EEG potentials predict upcoming emergency brakings during simulated driving. J Neural Eng 8:056001. https://doi.org/10.1088/1741-2560/8/5/056001
Haufe S, Kim J-W, Kim I-H et al (2014) Electrophysiology-based detection of emergency braking intention in real-world driving. J Neural Eng 11:056011. https://doi.org/10.1088/1741-2560/11/5/056011
Hayasaka S, Nichols TE (2004) Combining voxel intensity and cluster extent with permutation test framework. Neuroimage 23:54–63. https://doi.org/10.1016/j.neuroimage.2004.04.035
Hernández LG, Mozos OM, Ferrández JM, Antelis JM (2018) EEG-based detection of braking intention under different car driving conditions. Front Neuroinform 12:29. https://doi.org/10.3389/fninf.2018.00029
Hong K-S, Khan MJ (2017) Hybrid brain-computer interface techniques for improved classification accuracy and increased number of commands: a review. Front Neurorobot 11:35. https://doi.org/10.3389/fnbot.2017.00035
Jonsson S, Jonsson B (1975) Function of the muscles of the upper limb in car driving. Ergonomics 18:375–388. https://doi.org/10.1080/00140137508931471
Käthner I, Wriessnegger SC, Müller-Putz GR et al (2014) Effects of mental workload and fatigue on the P300, alpha and theta band power during operation of an ERP (P300) brain–computer interface. Biol Psychol 102:118–129. https://doi.org/10.1016/j.biopsycho.2014.07.014
Kasahara K, DaSalla CS, Honda M, Hanakawa T (2015) Neuroanatomical correlates of brain-computer interface performance. Neuroimage 110:95–100. https://doi.org/10.1016/j.neuroimage.2015.01.055
Khaliliardali Z, Chavarriaga R, Gheorghe LA, del Millán JR (2015) Action prediction based on anticipatory brain potentials during simulated driving. J Neural Eng 12:066006. https://doi.org/10.1088/1741-2560/12/6/066006
Kim J-W, Kim I-H, Lee S-W (2014) Decision of braking intensity during simulated driving based on analysis of neural correlates. 2014 IEEE international conference on systems, man, and cybernetics (SMC). IEEE, San Diego, pp 4129–4132
Kim I-H, Kim J-W, Haufe S, Lee S-W (2015) Detection of braking intention in diverse situations during simulated driving based on EEG feature combination. J Neural Eng 12:016001. https://doi.org/10.1088/1741-2560/12/1/016001
Kirchner EA, Tabie M, Seeland A (2014) Multimodal movement prediction—towards an individual assistance of patients. PLoS ONE 9:e85060. https://doi.org/10.1371/journal.pone.0085060
Krusienski DJ, Grosse-Wentrup M, Galán F et al (2011) Critical issues in state-of-the-art brain-computer interface signal processing. J Neural Eng 8:025002. https://doi.org/10.1088/1741-2560/8/2/025002
Lalitharatne TD, Teramoto K, Hayashi Y, Kiguchi K (2013) Towards hybrid EEG–EMG-based control approaches to be used in bio-robotics applications: current status, challenges and future directions. Paladyn, J Behav Robot 4:147–154. https://doi.org/10.2478/pjbr-2013-0009
Li H, Huang G, Lin Q et al (2018) Combining movement-related cortical potentials and event-related desynchronization to study movement preparation and execution. Front Neurol. https://doi.org/10.3389/fneur.2018.00822
Li J, Dimitrakopoulos GN, Thangavel P et al (2019a) What are spectral and spatial distributions of EEG–EMG correlations in overground walking? An exploratory study. IEEE Access 7:143935–143946. https://doi.org/10.1109/ACCESS.2019.2945602
Li Z, Zhang S, Pan J (2019b) Advances in hybrid brain–computer interfaces: principles, design, and applications. Comput Intell Neurosci 2019:e3807670. https://doi.org/10.1155/2019/3807670
Li J, Thakor N, Bezerianos A (2020) Brain functional connectivity in unconstrained walking with and without an exoskeleton. IEEE Trans Neural Syst Rehabil Eng 28:730–739. https://doi.org/10.1109/TNSRE.2020.2970015
Lin J, Liu S, Huang G et al (2018) The recognition of driving action based on EEG signals using wavelet-CSP algorithm. 2018 IEEE 23rd international conference on digital signal processing (DSP). IEEE, Shanghai, pp 1–5
Little S, Bonaiuto J, Barnes G, Bestmann S (2019) Human motor cortical beta bursts relate to movement planning and response errors. PLoS Biol 17:e3000479. https://doi.org/10.1371/journal.pbio.3000479
Liu Y, Ji X, Ryouhei H et al (2012) Function of shoulder muscles of driver in vehicle steering maneuver. Sci China Technol Sci 55:3445–3454. https://doi.org/10.1007/s11431-012-5045-9
Lloyd S (1982) Least squares quantization in PCM. IEEE Trans Inf Theory 28:129–137. https://doi.org/10.1109/TIT.1982.1056489
Lohani M, Payne BR, Strayer DL (2019) A review of psychophysiological measures to assess cognitive states in real-world driving. Front Hum Neurosci. https://doi.org/10.3389/fnhum.2019.00057
Lóopez-Larraz E, Birbaumer N, Ramos-Murguialday A (2018) A hybrid EEG–EMG BMI improves the detection of movement intention in cortical stroke patients with complete hand paralysis. 2018 40th annual international conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE, Honolulu, pp 2000–2003
Lotte F, Congedo M, Lécuyer A et al (2007) A review of classification algorithms for EEG-based brain-computer interfaces. J Neural Eng 4:R1–R13. https://doi.org/10.1088/1741-2560/4/2/R01
Lotte F, Bougrain L, Cichocki A et al (2018) A review of classification algorithms for EEG-based brain–computer interfaces: a 10 year update. J Neural Eng 15:031005. https://doi.org/10.1088/1741-2552/aab2f2
Ma J, Zhang Y, Cichocki A, Matsuno F (2015) A novel EOG/EEG hybrid human-machine interface adopting eye movements and ERPs: application to robot control. IEEE Trans Biomed Eng 62:876–889. https://doi.org/10.1109/TBME.2014.2369483
Mader M, Bresges A, Topal R et al (2009) Simulated car driving in fMRI—Cerebral activation patterns driving an unfamiliar and a familiar route. Neurosci Lett 464:222–227. https://doi.org/10.1016/j.neulet.2009.08.056
Makeig S, Debener S, Onton J, Delorme A (2004) Mining event-related brain dynamics. Trends Cogn Sci 8:204–210. https://doi.org/10.1016/j.tics.2004.03.008
Maris E, Oostenveld R (2007) Nonparametric statistical testing of EEG- and MEG-data. J Neurosci Methods 164:177–190. https://doi.org/10.1016/j.jneumeth.2007.03.024
Mewett DT, Nazeran H, Reynolds KJ (2001) Removing power line noise from recorded EMG. 2001 conference proceedings of the 23rd annual international conference of the IEEE Engineering in Medicine and Biology Society. IEEE, Honolulu, pp 2190–2193
Milosevic B, Benatti S, Farella E (2017) Design challenges for wearable EMG applications. Design, automation test in europe conference exhibition (DATE), 2017. IEEE, Lausanne, pp 1432–1437
Müller K-R, Krauledat M, Dornhege G et al (2004) Machine learning techniques for brain-computer interfaces. Biomed Eng 49:11–22
Neuper C, Wortz M, Pfurtscheller G (2006) ERD/ERS patterns reflecting sensorimotor activation and deactivation. Prog Brain Res 159:211–222. https://doi.org/10.1016/S0079-6123(06)59014-4
Nguyen T-H, Chung W-Y (2019) Detection of driver braking intention using EEG signals during simulated driving. Sensors 19:2863. https://doi.org/10.3390/s19132863
Novak D, Omlin X, Leins-Hess R, Riener R (2013) Predicting targets of human reaching motions using different sensing technologies. IEEE Trans Biomed Eng 60:2645–2654. https://doi.org/10.1109/TBME.2013.2262455
Oka N, Yoshino K, Yamamoto K et al (2015) Greater activity in the frontal cortex on left curves: a vector-based fNIRS study of left and right curve driving. PLoS ONE 10:e0127594. https://doi.org/10.1371/journal.pone.0127594
Pfurtscheller G, Lopes da Silva FH (1999) Event-related EEG/MEG synchronization and desynchronization: basic principles. Clin Neurophysiol 110:1842–1857. https://doi.org/10.1016/S1388-2457(99)00141-8
Pfurtscheller G, Neuper C (1994) Event-related synchronization of mu rhythm in the EEG over the cortical hand area in man. Neurosci Lett 174:93–96. https://doi.org/10.1016/0304-3940(94)90127-9
Pfurtscheller G, Neuper C, Andrew C, Edlinger G (1997) Foot and hand area mu rhythms. Int J Psychophysiol 26:121–135. https://doi.org/10.1016/S0167-8760(97)00760-5
Pfurtscheller G, Brunner C, Schlögl A, Lopes da Silva FH (2006) Mu rhythm (de)synchronization and EEG single-trial classification of different motor imagery tasks. Neuroimage 31:153–159. https://doi.org/10.1016/j.neuroimage.2005.12.003
Pfurtscheller G, Allison BZ, Brunner C et al (2010) The Hybrid BCI. Front Neurosci 4:30. https://doi.org/10.3389/fnpro.2010.00003
Pick AJ, Cole DJ (2006) Neuromuscular dynamics in the driver–vehicle system. Veh Syst Dyn 44:624–631. https://doi.org/10.1080/00423110600882704
Pineda JA (2005) The functional significance of mu rhythms: translating “seeing” and “hearing” into “doing.” Brain Res Rev 50:57–68. https://doi.org/10.1016/j.brainresrev.2005.04.005
Rendeiro C, Rhodes JS (2018) A new perspective of the hippocampus in the origin of exercise-brain interactions. Brain Struct Funct 223:2527–2545. https://doi.org/10.1007/s00429-018-1665-6
Rizzo R, Zhang X, Wang JWJL et al (2020) Network physiology of cortico–muscular interactions. Front Physiol. https://doi.org/10.3389/fphys.2020.558070
Rizzolatti G, Luppino G (2001) The cortical motor system. Neuron 31:889–901. https://doi.org/10.1016/S0896-6273(01)00423-8
Roland PE (1984) Organization of motor control by the normal human brain. Hum Neurobiol 2:205–216
Saha S, Baumert M (2020) Intra- and inter-subject variability in EEG-based sensorimotor brain computer interface: a review. Front Comput Neurosci 13:87. https://doi.org/10.3389/fncom.2019.00087
Saha S, Mamun KA, Ahmed K et al (2021) Progress in brain computer interface: challenges and opportunities. Front Syst Neurosci. https://doi.org/10.3389/fnsys.2021.578875
Sakihara K, Hirata M, Ebe K et al (2014) Cerebral oscillatory activity during simulated driving using MEG. Front Hum Neurosci. https://doi.org/10.3389/fnhum.2014.00975
Sarasola-Sanz A, Irastorza-Landa N, López-Larraz E et al (2017) A hybrid brain-machine interface based on EEG and EMG activity for the motor rehabilitation of stroke patients. 2017 International Conference on Rehabilitation Robotics (ICORR). IEEE, London, pp 895–900
Savić AM, Lontis ER, Mrachacz-Kersting N, Popović MB (2020) Dynamics of movement-related cortical potentials and sensorimotor oscillations during palmar grasp movements. Eur J Neurosci 51:1962–1970. https://doi.org/10.1111/ejn.14629
Salvador S, Chan P. "Determining the number of clusters/segments in hierarchical clustering/segmentation algorithms." 16th IEEE International Conference on Tools with Artificial Intelligence. 2004 pp. 576–584. https://doi.org/10.1109/ICTAI.2004.50
Schier MA (2000) Changes in EEG alpha power during simulated driving: a demonstration. Int J Psychophysiol 37:155–162. https://doi.org/10.1016/S0167-8760(00)00079-9
Schoffelen J-M, Gross J (2009) Source connectivity analysis with MEG and EEG. Hum Brain Mapp 30:1857–1865. https://doi.org/10.1002/hbm.20745
Schweizer TA, Kan K, Hung Y et al (2013) Brain activity during driving with distraction: an immersive fMRI study. Front Hum Neurosci. https://doi.org/10.3389/fnhum.2013.00053
Spiers HJ, Maguire EA (2007) Neural substrates of driving behaviour. Neuroimage 36:245–255. https://doi.org/10.1016/j.neuroimage.2007.02.032
Stancák AJ, Pfurtscheller G (1996) Mu-rhythm changes in brisk and slow self-paced finger movements. NeuroReport 7:1161–1164
Teng T, Bi L, Liu Y (2018) EEG-based detection of driver emergency braking intention for brain-controlled vehicles. IEEE Trans Intell Transp Syst 19:1766–1773. https://doi.org/10.1109/TITS.2017.2740427
Tortora S, Tonin L, Chisari C et al (2020) Hybrid human–machine interface for gait decoding through Bayesian fusion of EEG and EMG classifiers. Front Neurorobot. https://doi.org/10.3389/fnbot.2020.582728
Trigili E, Grazi L, Crea S et al (2019) Detection of movement onset using EMG signals for upper-limb exoskeletons in reaching tasks. J Neuroeng Rehabil 16:45. https://doi.org/10.1186/s12984-019-0512-1
Usakli AB, Gurkan S, Aloise F et al (2009) A hybrid platform based on EOG and EEG signals to restore communication for patients afflicted with progressive motor neuron diseases. Conference proceedings: annual international conference of the IEEE engineering in medicine and biology society IEEE Engineering in Medicine and Biology Society Annual Conference. IEEE, Minneapolis, pp 543–546. https://doi.org/10.1109/IEMBS.2009.5333742
Usakli AB, Gurkan S, Aloise F et al (2010) On the use of electrooculogram for efficient human computer interfaces. Comput Intell Neurosci. https://doi.org/10.1155/2010/135629
van Wijk BCM, Beek PJ, Daffertshofer A (2012) Neural synchrony within the motor system: what have we learned so far? Front Hum Neurosci 6:252. https://doi.org/10.3389/fnhum.2012.00252
Vecchiato G (2021) Hybrid systems to boost EEG-based real-time action decoding in car driving scenarios. Front Neuroergonom. https://doi.org/10.3389/fnrgo.2021.784827
Vecchiato G, Vecchio MD, Ascari L et al (2018) Electroencephalographic time-frequency patterns of braking and acceleration movement preparation in car driving simulation. Brain Res. https://doi.org/10.1016/j.brainres.2018.09.004
Vecchiato G, Del Vecchio M, Antopolskiy S et al (2020) Alpha and beta EEG desynchronizations anticipate steering actions in a driving simulation experiment. In: Ahram T, Karwowski W, Vergnano A et al (eds) Intelligent human systems integration 2020. Springer, Cham, pp 260–265
Walter H, Vetter SC, Grothe J et al (2001) The neural correlates of driving. NeuroReport 12:1763–1767
Wang W, Xi J, Zhao D (2018) Learning and inferring a driver’s braking action in car-following scenarios. IEEE Trans Veh Technol 67:3887–3899. https://doi.org/10.1109/TVT.2018.2793889
Wöhrle H, Tabie M, Kim SK et al (2017) A hybrid FPGA-based system for EEG- and EMG-based online movement prediction. Sensors 17:1552. https://doi.org/10.3390/s17071552
Zhang H, Chavarriaga R, Khaliliardali Z et al (2015) EEG-based decoding of error-related brain activity in a real-world driving task. J Neural Eng 12:066028. https://doi.org/10.1088/1741-2560/12/6/066028
Acknowledgements
The present study was supported by a research agreement between CAMLIN Limited and IN-CNR, and a research agreement between CAMLIN Limited and Toyota Motor Europe.
Funding
The research was supported by a research agreement between CAMLIN Limited and IN-CNR, and a research agreement between CAMLIN Limited and Toyota Motor Europe.
Author information
Authors and Affiliations
Contributions
All authors contributed to the study conception and design. Material preparation and data collection were performed by GV and MDV. Data analysis was performed by GV. The first draft of the manuscript was written by GV and all authors commented on previous versions of the manuscript. All authors contributed and approved the final manuscript.
Corresponding author
Ethics declarations
Conflict of interest
Financial interests: Author LA has received research support from Toyota Motor Europe. Author PA has received research support from CAMLIN Limited. Non-financial interests: none.
Ethical approval
The research was approved by the local Ethical Committee (comitato etico Unico per la Provincia di Parma).
Consent to participate
All subjects expressed written consent to participate in the research.
Consent for publication
All Authors agreed on the final version of the manuscript.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Vecchiato, G., Del Vecchio, M., Ambeck-Madsen, J. et al. EEG–EMG coupling as a hybrid method for steering detection in car driving settings. Cogn Neurodyn 16, 987–1002 (2022). https://doi.org/10.1007/s11571-021-09776-w
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11571-021-09776-w