Keywords

1 Introduction

Pupil dilation is an important metric for assessment of mental workload [15, 21], especially in a safety critical system [8] such as a helicopter (or) ship’s bridge or driving [40]. In these environments, the light condition fluctuates dramatically. Since changing light conditions can also impact pupil dilation, it is necessary to separate the effect of the mental workload from the effect of the changing light conditions to be able to utilise it reliably to evaluate the risk profile.

As discovered through the literature review, currently, there is no open and validated method to measure cognitive workload in a field condition trough pupillometry. The only commercially available method is proprietary [26] ad thus closed source, it disconnects the researcher from the ability to adapt the tools to specific research questions and have a deep understanding of the variables at play [17]. Moreover, no methods have been validated for use with low-cost eye trackers, which would enable affordable data gathering, including collaborative studies with multiple eye trackers.

The relation between cognitive workload and pupillary responses has been assessed as back as 1964 [15], here Hess and Poltmeasured changes in the pupil size of a subject during the resolution of “simple multiplication problems” and observed a link between the pupillary response and the difficulty level. Likewise, Kahneman et al. [21] investigated the correlation between task-evoked pupillary diameter and memory intensive tasks; reporting different pupillary responses from the learning and report/recollect phases as well as variations directly related to the task difficulty. These initials results were validated for a variety of “intensive cognitive tasks”, including language, writing, listening, speech and the solution of mathematical problems [22]. The psycho-physiological studies on workload and the pupillary response are often limited by three main factors:

  • Perceived luminance, as the variable with the more significant influence on the pupil diameter, it can mask the influence of cognitive workload. It is common to keep luminance as controlled conditions to isolate its effect.

  • Real-time tracking is only possible tough a limited number of the reviewed ed methods. Online tracking of cognitive workload, [42] and [26] requires high-temporal-resolution, as well as the control of the environmental variables, including the estimate of a continually changing baseline value. Lack of such features limits the study to the evaluation of well defined/separated tasks.

  • Open source. The only method that currently provides a solution to both the previous limitation is tied to patented technology, only limited documentation is available on the underlying method [26], and this makes it impossible for independent researchers to reuse, adapt and improve on such systems.

1.1 Research Questions

This paper is part of an ongoing, work-in-progress, research and will present only preliminary results to the following research questions:

  • How to reliably measure luminance from the POV of a subject using a small calibrated video camera?

  • How to calculate the baseline pupil size for a visual stimulus and use it to infer the cognitive state of the subject then?

  • How to scale this method in field conditions (where luminance variate in an unpredictable manner)?

1.2 Planned Contributions

  • Open source code, available in a public repository that can work on files generated by at least one common eye-tracker vendor.

  • Thesis report and paper with validation data.

1.3 Existing Research

Cognitive Workload. Mental workload can be evaluated through a variety of methodologies [5, 35]:

  • Subjective-empirical measures of perceived effort as rated by the subject.

  • Performance of the subject in a controlled task.

  • Physiological indices of cognitive state.

Subjective reports such as questionnaires and multidimensional ratings (e.g. the NASA-TLX [14]) are indirect means of evaluation of the perceived workload. They are usually easy and cheap to administer but have several limitations [39]. As post-facto evaluations, relying on the personal impression and memory of the subject, they do not track a change over time and are therefore difficult to use for the identification of specific peaks on cognitive workload. The NASA-TLX questionnaire is a standardised assessment tool of cognitive workload. It employees a “multi-dimensional rating scale” measuring six parameters to give an estimate of the overall task workload: mental, physical, temporal, frustration, performance and effort. Like other forms of self-report, it doesn’t record changes in cognitive load over time (multiple questionaries can be used to assist a complex task if divisible in subtasks). Other subjective workload measures are: Multiple Resources Questionnaire (MRQ) [2], Subjective Workload Assessment Technique (SWAT) [34], Overall Workload Level (OWL) [20] and Integrated Workload Scale (IWS) [31].

Performance-based measures of mental workload indirectly measure the cognitive state of a subject through the execution of a standardised task. Changes (speed, accuracy, response time) in the execution of the secondary task can be interpreted as a difference in cognitive/visual workload. The ISO defined Detection Response Task (DRT) [19] is an example of a performance-based method, Cegovnik et al. [4] used a tactile DRT to validate the use of a Tribe eye tracker to assess changes in the cognitive load of the subjects using oculography and pupillometry. The DTR estimates the cognitive load trough response rate and miss-rate of the response task: a stimulus is delivered through a vibrator attached to the subject in a random sequence; the test measures the response time (time needed to press a button attached to the steering wheel in a driving simulator). The use of this class of methodologies is to be planned considering the effect of the controls tasks on the main task as well as the low temporal resolution of the events that can be measured. Moreover, the relationship between cognitive workload and task performance is not linear and follows an inverted U-shape as defined in the Hebb-Yerkes-Dodson Law [4]. Both overload under-load can, therefore, result in decreased performances making the measure potentially unreliable if not paired with subjective data.

Physiological indices indirectly connect a measurable psychophysical parameter to an expected mental workload. Heart rate, respiratory rate, galvanic skin response, brain activity (EEG, fRMITCD), as well as eye activity [3] are parameters that over the years have been used to measure the mental workload. Modern eye tracking has an intrinsic advantage of being unobtrusive, and less impending that most of the other aforementioned techniques and could be a reliable instrument for over time monitoring of the mental workload [5]. Still, psychophysiological measures have several limitations [4] when applied in field conditions. In most of the studies mentioned beforehand, the experimental design included two or more tasks with different levels of difficulty indirectly estimating the workload required for each of the tasks. The physiological and neurological models employed in the psychophysiological methods have to be specially designed and trained to fit a particular task-evoked neural activity. It is, therefore, difficult to compare the results to a generalised measure of workload.

Blinks. Eyeblink duration and rate have been identified as an alternative metric for visual workload [1] and [33]. Unfortunately, this metric reliability is limited, as the blink rate can be influenced, in an opposite manner, by both the mental workload and visual workload.

Pupil. In the Handbook of Psychophysiology [3, p. 443], Cacioppo et al. defines the pupillary system as a “dually innervated organ”. The pupil size is determined by the concurring action of a parasympathetically innervated constricting muscles and sympathetically innervated radial dilator muscles. The parasympathetic activity is dominant, responding to light reflexes, and determine the varying pupil size baseline, the sympathetic activity is instead connected to behavioural and stress contexts and can be used as a psychophysiological parameter of cognitive activity.

Task-Evoked Pupillary Response. Palinko et al. [30] estimated the driver’s cognitive load from pupil size measurements finding that it the pupillary response correlates with the measured driving performances, and this as similar studies seems to confirm the reliability of pupillometry as a measure of cognitive workload. However, the analysis is limited to a simulated task with low variability between target luminance. Palinko et al. [30] introduced a pupillometric cognitive load measure for real-time cognitive load changes (every several seconds).

Light and Cognitive Load Effects on Pupil Diameter. Palinko et al. [29] follows up the previous study with a proof of concept of the possible separation between cognitive and ambient light components of pupil dilatation. The study was conducted using a driving simulator as a controlled environment for both an Aural Vigilance Task an Illumination Task (with different brightness targets) and a combined validation task. The study results show that it should be theoretically possible to model the psychophysical functions of the pupillary response over time to light stimuli and shows the measured trend over time. It also notes how the transitions bright/dark/bright are not equal as different muscle groups are involved in the contraction and dilation movements. The bright light reaction is quick “to protect the retina from overexposure”, while the reaction to darkness is slower and gradual. The psychophysical function to predict an expected baseline pupil diameter should, therefore, take into account multiple parameters, current light level, previous light level, the rate of change, as well as age, and target. The study concludes that it is possible to discern the effects of luminance and cognitive load on pupil diameter and that the “proof of concept” predictor works in the limited experimental setting.

Discern Between Mentally and Visually Workload. Recarte et al. [32] validated the use of pupillary response as workload index in a field scenario ignoring the effect of illumination changes as the variable was impossible to control. The data they collected shows consistent results across the different driving task (no task, verbal task and visual task) collected during multiple driving session, to such a degree that cannot be explained by the sole different lighting conditions.

Unified Formula for Light-Adapted Pupil Size. Since the pupil diameter can be deconstructed as the result of multiple concurring factors, in order to correctly differentiate the cognitive workload from the pupillary light response, it can be useful to compute the expected pupil diameter for a given brightness condition and use the resulting value as a baseline value upon which calculate the cognitive-driven component of the measured pupil size. In a recent paper, Watson et al. [41] (NASA Ames Research Center and University of California) have reviewed seven different published psychophysical functions defining the relation between target luminance (cd/m\(^2\)) and expected pupil diameter. In the same paper, they also published a newly developed unified formula. The calculated baseline would work in the range of 2 to 8 mm, the reliability of the unified formula have to be tested to ensure that the little variability rage of the pupil size provoked by cognitive workload is preserved (\({<}{\pm }1\) mm) [29]. The unified formula [41] is valid only for a light-adapted condition with stable illuminant and point of view (PoV) as it doesn’t account for the adaptation state or the “pupillary unrest” (low-frequency random fluctuation in the range of 0.02 to 2.0 Hz and amplitude within ±0.25 mm).

Independent variables in the unified equation:

  • Luminance.

  • Age (The maximum pupil diameter, as well as total range, declines as the age grows).

  • Field diameter (deg).

  • Number of eyes stimulated, the final diameter is dependent on the number of eyes that are adapted to the light condition they defined the “Effective corneal flux density” (the variable controlling the effective pupil diameter) as dependent on the number of eyes (attenuated by a factor of 10 for one eye).

$$\begin{aligned} F = LaM(e) \end{aligned}$$

(F) Flux density as the product of (L) luminance, (a) area, and (M(e)) monocular effect.

Wavelet Analysis. Marshall et al. [26] describes a technique to identify the origin of a recorded pupillary response that works independently from the target luminance. The procedure employs wavelet analysis to identify the dilation reflexes of the subject’s pupil. She explains how the reflexes can be differentiated as the pupil have different responses to light and psychosensorial stimulus. In a steady light, the pupil shows an irregular pulsation (light reflex) provoked by the interaction of the circular contracting muscles (agonist) and antagonist radial muscles act as the antagonist and are inhibited from dilating the pupil. A cognitive workload provokes a different waveform as both circular and radial muscles dilate the pupil creating a brief peak. This would imply that the cognitive workload is measured as the frequency and intensity of such events and not as a steady dilatation of the pupil (for the duration of the load), but it is unclear how this method would perform in a field condition, with highly variable ambient luminance.

An application of this technology is explained in a study conducted by Marshall for the US Navy. She applied the patented metric of Index of Cognitive Activity (ICA) [27] thanks to a networked system that is set up to record the cognitive workload for multiple team members during a collaborative task. The study assessed the performance of a three-person team in a simulation system, and the effort to overcome mission-related problems. A similar study, a collaboration between NASA Ames and EyeTracking, Inc. used the ICA and eye metrics to detect the difference between low and high fatigue states [28].

Machine Learning for Pupillometry. Wierda et al. [42] and the related work of Ferscha et al. [9] represent a different approach to the problem of the indirect assessment of mental workload. As the response time of the pupil to a mental workload event is too slow (several seconds) to be used as a real-time measure, it can be used directly only as an average over time. This makes it suitable to evaluate lengthy tasks that have a reasonably constant load in cognitive workload (at least several seconds). These two studies show a proof of concept of how to obtain an high-temporal-resolution (c.a. 10 Hz) tracking of the cognitive processes through deconvolution. The aim of real-time cognitive workload measurement gains value in the context of the implementation of a real-time feedback loop in the interaction design of a system (e.g. a system able to respond to different cognitive states of the user).

Wierda et al. [42] fixed the distribution of “attention impulses” every 100 ms defining the output’s temporal resolution. Employing a model of the “Task-evoked pupil impulse response,” it reconstructs the intensity of the attention impulse that provoked the measured pupillary response. Ferscha et al. [9] further developed the concept through machine learning for better performances without the need of a fixed temporal resolution of the cognitive impulses. To reduce the effect of incident light Ferscha et al. [9] used the average illumination in the subject’s field of view analysing the eye tracker camera stream. The technique they used is possibly still insufficient to adapt the technology to a field study with a highly variable illumination. In the described implementation a luminance change more significant than the set threshold would trigger a suspension of the tracking, this state is then maintained until the condition is stable again and a new baseline can be calculated. A similar solution was implemented to filter out blinks. The dynamic baseline is computed through a series of threshold and doesn’t adjust for small changes in target luminance.

Illuminance Measurement Using a Digital Camera. Luminance as a measure needed to dynamically estimate the pupil size of a subject, candela per square meter cd/m\(^2\), is the quantity of light radiating from a source. An illuminance meter is an expensive and bulky device, in more than one instance this has resulted in attempts to use a camera as cheaper and more flexible alternative [16] and [43].

A digital sensor is at its core an array of Illuminance sensors. Each pixel measures the number of photons hitting the photoelectric surface. The presence of a Bayer filter for colour photography makes it so that to reconstruct the information form the entire visible spectrum multiple pixels have to be analysed at the same time. Each pixel in the final image has reconstructed values from the neighbouring pixels for all the RGB channels and in itself would be sufficient to reconstruct the illuminance of the scene, in order to reduce noise and gain reliability multiple pixels should be used, the number of pixels used is effectively the field of view of the instrument [16]. The formula proposed by Hiscocks [16] has been optimised for a DSLR camera, not all the parameters are accessible when using an embedded digital video camera.

Parameters:

  • Pixel value (0–255 for an 8bit monochrome image).

  • Shutter Speed (In a video camera, this is limited by the frame rate, e.g. 1/30 s for a 30 fps camera) and aperture or focal ratio.

  • Iso or film speed.

  • Camera Constant (The calibration constant for a specific camera model that has to be determined with a known instrument).

Wuller [43] suggest a different model, reversing the colour processing of the camera, first from gamma-compressed RGB to linear RGB and then from linear RGB to CIE XYZ, extrapolating then \(y( \lambda ) \) as the relative luminance (luminance as defined by the luminosity function, reproducing the spectral luminous efficiency of the human eye). The author notes that to access the linear response of an image sensor the correct inverse gamma has to be applied and that this could deviate from the standard 2.2 (sRGB), the relative luminance can then be converted to luminance through a linear relation specific for a particular sensor/camera settings combination.

Maritime Usability and SA. Endsley et al. [6] defines situational awareness (SA) as “the perception of the elements in the environment within a volume of space and time, the comprehension of their meaning and the projection of their status in the near future”. Low SA has been found to be one of the primary sources of human error in safety-critical systems [36]. Real-time monitoring of SA seems possible through the analysis of the subject visual attention aided by a variety of eye tracking data such as:

  • Fixation duration: length of fixations (e.g. time spent on a single target without movement).

  • Fixation rate: average number of fixations in a unit of time.

  • Dwell time: the sum of all the fixation time in a single area of interest.

  • Saccadic main sequence: the relation between the saccadic duration and magnitude and between peak velocity (PV) and magnitude [5], as both PV and duration increase with the magnitude.

    • Saccadic duration: the period between two positions of the fovea.

    • Saccadic magnitude: the magnitude of the saccadic movement (angle).

    • Peak saccadic velocity: highest velocity reached during saccades deg/sec.

Use of SAGAT. The Situation Awareness Global Assessment Technique (SAGAT), is a global tool developed to assess SA [7]. “A simulation employing a system of interest is frozen at randomly selected times, and operators are queried as to their perceptions of the situation at that time. The system displays are blanked, and the simulation is suspended while subjects quickly answer questions about their current perceptions of the situation. As a global measure, SAGAT includes queries about all operator SA requirements, including Level 1 (perception of data), Level 2 (comprehension of the meaning) and Level 3 (projection of the near future) components. This includes a consideration of system functioning and status, as well as relevant features of the external environment. SAGAT queries allow for detailed information about subject SA to be collected on an element by element basis that can be evaluated against reality, thus providing an objective assessment of operator SA.”

Ikuma et al. [18] compared different standard human factors measurement tools: workload ratings (SWAT and NASA-TLX) and Situation Awareness Global Assessment Technique (SAGAT). Eye tracking was also used to analyse the gaze path of the participants during the simulation, “the percentage of time spent looking at different areas of the screen during steady-state periods differed among workload levels”. This study only looks at a small number of areas of interest (AOIs) on the interface, to infer the visual attention of the subjects for different areas of the interface. The usability of on-board interfaces on High-Speed Craft (HSC) has been assessed through the application of eye tracking technology [11]. The cognitive workload and SA of the crew of a military HSC in littoral waters were selected as of interest because of the combination of high-speed navigation and the need to navigate outside established routes. With particular interest on the role of the navigator [10] and its use of the onboard interface “Route Monitor Window”. Hareide et al. [11] collected data from both field and simulator activities using the Tobii Pro Glasses 2 Eye Tracker. The methodology of the study tried to account for the difference in the environment and datasets between the simulator and field conditions. The Author followed up a mid-life update of the interface [12, 13], with further validation of the redesigned interface for the primary objective of increased navigator attention dedicated to the “outside” Area of interest opposed to the various interfaces. Eye trackers were in this case used as to indirectly evaluate situational awareness of the navigator through quantisation of the time spent on the interface rather than observing the environment. Hareide et al. [12] apply the concept of dwell time, look-backs and Backtracks to the analysis of AOIs:

  • Look-backs (returns, refixation) are saccades landing in an AOIs already visited. The analysis of a look-back can point to a variety of concurring factors: memory failure, confusion on the function of a command/element, the difficulty of content understanding and intrinsic importance of the information present in an AOI.

  • Backtracks are calculated on the specific sequence of saccades and are a sudden (inverted gaze direction) rapid eye movement back to a just visited AOI. Confusion or uncertainty, changes in goals, a mismatch between the users’ expectation and interface layout.

The author shows how eye tracking data can be used to guide the development of a GUI through the analysis of areas of interest and gaze behaviour, but also notes several limitations in the use of the eye tracker that need to be considered not to influence the behaviour of the user group. This includes the thickness of the eyepiece frame, creating a visible “frame of vision”, unwanted reflection and glares on the protective glass, difficulty using the binoculars in conjunction with the trackers and unfavourable lighting conditions.

2 Methods

2.1 Measure of Luminance from POV

This is a work in progress. his research starts with the development of the necessary tools. Even though it is theoretically possible to use a video camera as luminance meter, the reliability and accuracy of this technique will depend significantly on the software and hardware. The calibration and validation of the equipment will be done with a know good instrument in the Norwegian Colour and Visual Computing Laboratory. A colour checker will be measured with both the Konica Minolta CS-2000 [25] spectroradiometer and the World Camera mounted on the Pupil lab eye tracker (Pupil Pro). The data can then be used to calibrate the World Camera as a rudimentary luminance meter. A colour checker illuminated by a diffuse light at a variable intensity will be measured through both the Pupil Pro and the spectroradiometer. Different combinations of illumination and exposure settings on the software are required to model the sensor response.

The Pupil lab [23] software in his current version (1.10.20) does not dynamically save the exposure settings during the recording. Libuvc [38] (cross-platform library for USB video devices) is used to receive the video stream and communicate with the two cameras. Libuvc supports either getting or setting the exposure value and should allow retrieving the current exposure data during the recording. The lack of support of these functionalities in the Pupil lab software makes it impossible to use automatic exposure as the calibration values would change during the recording in an unpredictable manner. Using a fixed manual exposure is possible but severely limits the maximum dynamic range of the light meter.

Alternatives to the use of the camera, to simulate a scenario in which the aforementioned limitation does not apply, which would be easily reachable with some interest from the developers, is to use an external light meter mounted on the eye-tracker. This would provide a measurement that is not bound to the limited dynamic range of the camera with the drawback of having two disconnected data streams.

Instrumentation. The Konica Minolta CS-2000 spectroradiometer [25] is a high precision polychromatortype spectroradiometer, it will allow measurement on a vast range of luminance (0.3 to 500,000 cd/m\(^2\)) with a \(\pm 2\%\) accuracy. The Pupil Pro [23] and [24] World Camera is mounted just above the subject eye, facing outward. The camera offers different combinations of resolution and frame-rate \(1920 \times 1080\) @30 fps, \(1280 \times 720\) @60 fps, \(640\times 480\) @120 fps covering a FoV of 60 or 100\(^\circ \) diagonally (depending on the lens).

2.2 Calculate the Baseline Pupil Size

The measure of luminance from the POV will be connected to the unified formula for light-adapted pupil size developed by [41]. Two elements will need validation:

  • the accuracy and precision of the unified formula.

  • the different possible methods to convert the input from the camera to the correct input values for the formula.

The [41] unified formula is based on a standard procedure involving a defined stimulus: the observer is shown a bright circle on a dark background. The size (degrees of field of view) and luminance (cd/m\(^2\)) of the circle determine the corneal flux density (i.e. the product of luminance and subtended area) as defined by [37]: \(D = 7.75-5.75 [(F/846)0.41/((F/846)0.41 + 2)]\) “where D is the pupil diameter (mm), and F is the corneal flux density (cdm-2deg2)”.

The model implies that at its core the pupil control mechanism reacts as a ‘flux integrator’, following an S-shaped curve.

An image from the camera has to be used to evaluate the flux density or to indirectly convert the image into a corresponding standardised stimulus (i.e. a circle on a dark background). The most promising approach is to consider the average luminance on the camera sensor (or external light sensor) as equal to the luminance of a standardised stimulus (bright circle) as wide as the entire field of view (FoV) (120–190\(^\circ \)).

This assumption ties the precision of the calculated pupil size to how well the camera FoV matches the user FoV). To test the quality of the model NTNU students and staff (age from 20 to 50) will be recruited for validation in a controlled environment.

Validation. The procedure will refer to the methods used by Palinko et al. [29]. It will be divided into three parts:

  • No-load - variable light-adapted state. The participants will be sitting in a dark room looking at a selection of projected images; the sitting position will be adjusted to maintain a constant field of view and distance from the screen. The projected images will include standardised stimuli as well as more complex images (e.g. outdoor naturalistic scenery). Each image will be represented for several seconds to let the pupil reach an adapted state (c.a. 15 s). Each image will include a focal point ad the participants will be asked to stare at the focal point.

  • Load - static light-adapted state. With a constant ambient luminance (e.g. grey image projected), the participants will be asked to perform an Aural Vigilance Task (AVT), as in [29]. The task involves listening to a voice counting from 1 to 18, repeated multiple times. Every 6th number (6, 12, and 18) might contain errors (i.e. another number is replaced to the correct sequence). The participants would have to perform an action such as pressing a button when they detect an error. The task should induce an increased cognitive workload near every 6th number. The location of the error should be randomly selected for each session.

  • Load - variable light-adapted state. The same AVT task is repeated but with variable standardised images being projected.

2.3 Measure of Luminance from POV

Subjects. The subjects for the final session will be recruited as cadets of the Royal Norwegian Naval Academy (RNoNA) and will require access to the training vessel (Kvarven). The crew of a training vessel includes navigator, assistant, helmsman and training instructor. During a study session, the vessel would also include one to three researchers to set up and record the data. Depending on the availability of multiple eye trackers both the navigator and helmsman could participate in the experiment for each session.

Procedure. The setup for each session will include:

  • Introduction to the research and signature of the informed consent.

  • Application of the eye tracker (glasses and recording device).

  • Calibration of the eye tracker.

  • Reference measurements of ambient illumination.

  • The debriefing will include a short interview and the NASA-TLX questionnaire as a further reference of the cognitive workload.

Two researchers should be present on board at any time. Between each session, up to 30 min will be required for cleaning of the instrumentation, download of the test data and recharge of the various batteries.

The task aims to highlight different levels of cognitive workload. The navigation task should be repeatable Fig. 1, and it should last less than one hour (not including the setup and debriefing) and should include a mix of low and high workload for the subjects: E.g. Steady navigation - change of course - steady navigation.

Fig. 1.
figure 1

The course suggested by Odd Sveinung Hareid from Laksevåg (Bergen)

3 Expected Results

3.1 Proof of Concept

This is a work in progress, at the time of writing, the research is still in the initial exploratory phase and should be completed by the end of May 2019. The literature review helped in defining a path to follow in order to develop the necessary tools (the pupil baseline calculation), but there are inherent risks and challenges in the generalisation of findings that were initially only meant for controlled condition/laboratory study. A series of non validated tests is being carried out to determine:

  • weather the camera-based luminance meter works well enough for a reliable field application (limited dynamic range of the camera, limited bit depth, the difference in the camera FOV compared to the subject FOV).

  • Weather pupil baseline calculation is precise to such a degree not to mask the cognitive workload.

  • Weather the chosen eye tracker can operate in field conditions.

Fig. 2.
figure 2

This sample data output was recorded from a user sitting in front of a laptop in a dark room. The green line is the pupil size (mm) as measured by the eye tracker; the red line is the baseline as calculated from the video data and the blue line is the difference between the two. The blue will ultimately represent the cognitive workload. In this sample, the middle “steady” part has been measured on a subject performing an IQ test. The horizontal axe, time, is expressed in video frames at 30 fps. (Color figure online)

A series of artefacts have been identified in the sample data collected:

  • Pupillary overshoot, the model of the pupil size is specific to adapted state and doesn’t account for the pupil natural overshoot that can be observed when a rapid change in luminance occurs Fig. 2.

  • Pupillary unrest, in the form of low-frequency random fluctuation in the range of 0.02 to 2.0 Hz and amplitude within ±0.25 mm Figs. 2 and 3.

  • Incorrect measured pupil range; the pupil size is calculated from the video image in pixels to an estimated mm by the 3d model. different calibration of the pupil camera (distance from the eye) can bring the measured range outside the unified formula unified formula [41] range (c.a. more than 2 mm and less than 8 mm).

  • Luminance outside the camera dynamic range, this is visible in the second plot, in this case, an outdoor recording ended inside a building, the completely black image from the video produces an unreliable luminance reading Fig. 3.

Fig. 3.
figure 3

This outdoor session shows the limitations of the camera if used with a fixed exposure.

During the test multiple data sources will be combined including: Heart rate variability, POV camera, eye tracking camera all will be used to substantiate the measure of cognitive workload. Dates and travel The number of sessions depends on the availability of cadets, five to ten subjects would be a good result. Given the nature of the experiment weather and ambient illumination conditions should be kept constant within a reasonable range. This could require the spread of the study over multiple days.

To account for the limitations of the eye tracker, it would be advisable to plan a portion of the sessions after dawn, (lower the contrast between the user interface inside the cabin and the outside).

4 Conclusion

The development of the necessary tools is in progress and will hopefully end as a refined proof of concept and validation of the method with the intent of attracting the interest of developers to consolidate the application. The validation that will be attempted as part of the research will be by no mean be exhaustive, it is expected that the interest surrounding the measure of the cognitive workload will result in a variety of experiments on the topic, to further explore the benefits and limitations of the developed methods.

The choice of a camera as a luminance meter could severely limit the accuracy of the method but would allow a variety of head-mounted eye trackers to be used for cognitive workload studies without the need of any extra hardware.