Maritime Cognitive Workload Assessment
- First Online:
- Cite this paper as:
- Miklody D., Uitterhoeve W.M., van Heel D., Klinkenberg K., Blankertz B. (2017) Maritime Cognitive Workload Assessment. In: Gamberini L., Spagnolli A., Jacucci G., Blankertz B., Freeman J. (eds) Symbiotic Interaction. Symbiotic 2016. Lecture Notes in Computer Science, vol 9961. Springer, Cham
The human factor plays the key role for safety in many industrial and civil every-day operations in our technologized world. Human failure is more likely to cause accidents than technical failure, e.g. in the challenging job of tugboat captains. Here, cognitive workload is crucial, as its excess is a main cause of dangerous situations and accidents while being highly participant and situation dependent. However, knowing the captain’s level of workload can help to improve man-machine interaction. The main contributions of this paper is a successful workload indication and a transfer of cognitive workload knowledge from laboratory to realistic settings.
KeywordsWorkload BCI EEG
In the maritime world, as in many other workplaces, working memory, the ability to process information and to take decision is crucial. The quantification of cognitive workload is a measure to study these aspects. The insight can shed light onto the limitations posed by the human factor and point out how to improve equipment, conditions or training. While there exists no generally accepted definition of cognitive workload, there is a large agreement that it encompasses the two concepts of activation and resources or capacity .
In the present investigation, we designed a BCI system based on spectral decompositions of electroencephalographic data  that is trained to detect states of high/low cognitive workload. The workload was manipulated by the main task itself within one of the conditions while in the other we employed the 2-back task  as a secondary task which is a common tool for the measurement of workload . The aim of this study was to investigate cognitive workload in a more realistic environment and set the results into context with those from experiments obtained in clean laboratory settings. Therefore, professional tugboat captains were observed in a realistic training simulator study, where we investigated whether and how laboratory based cognitive workload studies are transferable to more realistic settings. In particular, we investigated if and how alpha & theta oscillations are modulated by cognitive workload.
1.1 Neurophysilogical Correlates of Cognitive Workload
Cognitive workload is reflected in different components of brain activity. In view of the present target application, modulations of event-related potentials due to workload [11, 12, 18] are not relevant, since there are no controlled and continuously repeated stimuli. Therefore, we concentrate on workload-induced modulations of spontaneous brain activity.
With respect to the more prominent alpha frequency band, most studies report a negative correlation of cognitive workload and alpha power at parito-occipital scalp locations, see e.g. [4, 7]. However, these studies used tasks in the visual modality to induce workload, such that one can only derive the implication of alpha reduction for workload in visual resources. In general, the functional role of alpha band oscillations is not yet conclusive. For a memory task in the auditory domain,  reports a modulation of theta oscillations only, but no modulations of the alpha rhythm. Some studies using auditory stimulation even found an increase of alpha activity with increasing workload [2, 10, 13, 16]. A possible interpretation is provided by the hypothesis of functional inhibition, which postulates that strong alpha activity reflects active inhibition of task-irrelevant processes : when the critical processing load is in the non-visual, the visual areas are actively deactivated. The idea to build EEG-based workload monitoring systems was presented, e.g. in [3, 4, 19].
2 Experimental Design
In a 10-participant simulator study, we recorded electroencephalographic data from a realistic tugboat scenario with professional captains (participant 8 excl.: sickness). The participants were recruited along the training network of MARIN and were compensated for their voluntary participation. They were all male with ages ranging from 30 to 65 years and different levels of experience. The experiment consisted of 3 different scenarios (approx. 40 min each), where scenario 1&3 were identical, see Fig. 1.
2.1 Phase 1 and 3: Bow-to-Bow
Condition 1: Free Sailing Condition: low workload: The captain was instructed to follow a large container ship astern while the weather was manipulated to have no extra effect on the workload.
Condition 2: Connecting Condition: high1 workload: After a transition phase moving to the front of the vessel the tugboat captain got the instruction to get ready for bow-to-bow connection while the weather conditions were changed to harsh (wind, waves, fog). Then, the captains were told to wait for connection for 5 min which represents the high1 phase.
Condition 3: Pulling Condition: high2 workload: After the rope was connected, a constant tow force and line length was instructed. The weather stayed harsh.
2.2 Phase 2: n-Back
The n-back task is commonly used in neuroscientific research as a manipulation tool for cognitive workload. We used this as a secondary task to have a condition comparable to common research and to see how much our bow-to-bow scenario corresponds to the neural patterns of this commonly known task. We used an auditory 2-back task, where the participant had to follow a stream of spoken numbers. If the last number heard corresponded with the digit 2 back, they had to press a button. The digits 1–9 were used with 3 s interleave randomly (75%) and forced 2-back repetition (25%) to get a reasonable amount of repetitions. The 2-back was played auditorily to keep a realistic behavioral scheme of the captain. There were 2 conditions, 4 min each, which were repeated 5 times, resulting in a total duration of 40 min for the whole phase (see also Fig. 3 on the right):
Condition 1: Free Sailing Condition low workload: In this condition, the same low workload task of the bow-to-bow Scenario was induced for comparison.
Condition 2: Free Sailing with 2-Back Condition high workload: The 2-back task was used additionally to the Free Sailing to induce a higher workload while keeping the primary task constant.
3.1 Dealing with Artifacts
A preliminary analysis of the data’s spatio-spectral content showed that the EEG of some participants was heavily affected by artifacts. This was expected due to the participants being allowed to act naturally. Head and trunk movements were required for the sailing tasks, as the simulator provided a 360\(^\circ \) projection.
First, the automatic artifact removal method MARA  had been employed, that gives good results in usual EEG datasets. The method is based on a decomposition of the multivariate EEG by the use of an Independent Component Analysis (ICA). The components were classified into artifacts and neuronal components. Then, the cleaned EEG signals were obtained by projecting only the neuronal components back into the sensor space. The classifier that distinguishes between artifactual and neuronal components was trained on a large data base of EEG datasets for which the ICA decomposition was manually annotated. For datasets that contain artifacts unlike those ones contained in the training data base, some of the artifactual components may go undetected. This seemed to be the case for the dataset at hand.
The grand average (i.e., average across participants) of the spectral analysis of the automatic artifact removal method MARA cleaned signals is shown in Fig. 5 (2-back task) and Fig. 6 (bow-to-bow task). When comparing the results for individual participants, it became clear that the grand average is only of limited use. The effect of the workload conditions seen in the alpha band varies with respect to the specific frequency range: we observed individual frequencies and topologies at which alpha levels appeared characteristic for the individual participants - also across conditions.
For the 2-back as a secondary tasks, effects in higher visual alpha can be observed across participants around parietal to occipital electrodes. The effect is contrary to common results in laboratory settings, as the alpha increases with increasing task difficulty. On a single participant level, we find this at an individual frequency of around 10–12 Hz in 6 of 7 participants (subj 1 excluded due to recording error, subj 10 excluded due to obvious task misunderstanding: button press after every stimulus). One participant with seemingly different results shows no effect at these frequencies but contrary signed \(r^2\) at lower alpha (around 9 Hz). A possible explanation to this difference is personal stress we observed on a subjective behavioral level. Unexpectedly, theta is not very relevant except for one participant.
4 Classification Analysis
4.1 Aim and Approach
The spectral analysis showed strong artifacts in the data. Apart from noise of the technical devices, there are artifacts from muscle activity (seen in high frequencies, mostly at outer temporal and occipital electrodes) as well as from eye movements (seen in low frequencies at very frontal channels). Furthermore, there may be motion artifacts due to the motion of the electrode cables induced by head and trunk movements.
4.2 Preprocessing, Artifact Reduction, Feature Extraction and Classification
We used 1 Hz high-pass filtering alone (R), in combination with the automatic ICA artifact reduction MARA  (C) as well as manual ICA artifact reduction (CM) (for details see Sect. 3.1). The blocks were subdivided into epochs of 1 min. Then, we built different spectral band power based features. In addition, we performed widely used Common Spatial Pattern analysis (CSP)  in different band combinations with the logarithm of the variances as features. We evaluated the different classification designs within phases as block-wise cross-validations (CV) as well as between phases to test for generalization. The classifier itself was based on regularized shrinkage linear discriminant analysis (rsLDA) .
The results of the 2-back scenario already show the complexity of a physiological index of cognitive workload, when the task is not performed in a constrained laboratory setting, but embedded into a more realistic and complex scenario. The expected increase of the frontal theta oscillation was in general not observed, and the power in the parietal alpha did not decrease as found in most workload studies, but it showed a contrary effect. The alpha effect is consistent with some of the literature on non-visual tasks. The common hypothesis for this ambiguity is that the task-irrelevant visual brain region is actively inhibited in order to focus resources to the relevant non-visual processing. One participant who showed a parieto-occipital alpha decrease might have had a visual strategy to memorize the sequence of numbers - albeit presented auditorily.
Classification of workload levels in the complex bow-to-bow scenario is in general successful. Transferring the classifier between the two phases 1 and 3 does not degrade the performance appreciably. The fact that classification in the realistic bow-to-bow task worked less well compared to the 2-back task requires consideration. As found in the literature on electrophysiological correlates of workload, there are two opposing effects concerning the modulation of the alpha rhythm. Tasks in the visual modality mostly decrease alpha activity, while workload in non-visual modalities might increase it (as in our 2-back paradigm). In the complex bow-to-bow scenario, these effects may be in conflict. Retrieving expertise about the maneuvering can be expected to be mainly non-visual. Nevertheless, the control of the boat does not allow a rigorous visual inhibition as it requires synchronized visual processing, in particular as the weather conditions were challenging during the high workload condition. This conflict can be assumed to lead to a much weaker effect on alpha power - an issue well worth deeper investigation.
Another interesting point in the view of applicability is the fact that the complex concept workload encompasses different factors, two of which are activation and resources. Therefore, a high output of the workload monitor could indicate a strong activation in the sense of an effective focusing of the task at hand (inhibiting task-irrelevant processing). Accordingly, an interpretation of the participant being at the limit of her/his resources might not be adequate.
Out-of-the-lab results are not easily transferable to realistic settings, but individual strategy and modality dependent neural activity can be used to adopt the interaction between system and user towards a symbiosis.
This research was mainly funded by MARIN, The Netherlands and we want to thank MARIN’s Nautical Center, Wageningen for the very kind collaboration. The research leading to these results has also received funding from the European Union Seventh Framework Programme (FP7/2007-2013) under grant agreement no 611570 (MindSee), and by the BMBF (contract 01GQ0850, BFNT).
Open Access This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.
The images or other third party material in this chapter are included in the chapter’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.