Modeling cognitive load and physiological arousal through pupil diameter and heart rate
- 176 Downloads
This study investigates individuals’ cognitive load processing abilities while engaged on a decision-making task in serious games, to explore how a substantial cognitive load dominates over the physiological arousal effect on pupil diameter. A serious game was presented to the participants, which displayed the on–line biofeedback based on physiological measurements of arousal. In such dynamic decision-making environment, the pupil diameter was analyzed in relation to the heart rate, to evaluate if the former could be a useful measure of cognitive abilities of individuals. As pupil might reflect both cognitive activity and physiological arousal, the pupillary response will show an arousal effect only when the cognitive demands of the situation are minimal. Evidence shows that in a situation where a substantial level of cognitive activity is required, only that activity will be observable on the pupil diameter, dominating over the physiological arousal effect indicated by the pupillary response. It is suggested that it might be possible to design serious games tailored to the cognitive abilities of an individual player, using the proposed physiological measurements to observe the moment when such dominance occurs.
KeywordsSerious games Physiology Electrocardiogram Pupil diameter Arousal Cognitive load
Research shows that emotions impair or facilitate decision-making [1, 19]. Russell’s model  classifies emotions through valence and arousal, their independent components. Therefore, valence represents whether the current emotional experience of a situation is positive or negative. On another hand, the level of excitement is represented by arousal. Following this classification, emotions are measured through the valence and arousal components. The concept of physiological arousal has been generally validated in models of emotion . Task difficulty is correlated with physiological arousal , thus a challenging decision task might be reflected in physiological signals. The author further states that there is a moment when performance decreases as arousal continues to increase. The pupil dilation is influenced by physiological arousal, together with attention and interest [3, 5]. The pupil diameter (PD) has been validated as a useful measure of physiological arousal [15, 23]. Early investigators  viewed the pupillary response as reflecting the level of arousal or emotionality; followed a few years later [18, 22] by studies arguing that the pupillary response might reflect cognitive activity as well as or instead of arousal. Thus, Stanners et al.  concludes that the pupil response will show an arousal effect only when the cognitive demands of the situation are minimal. If the situation requires a substantial level of cognitive activity, only that activity will be observable on the pupil diameter, dominating over the physiological arousal effect indicated by the pupillary response . These findings give rise to a need for the investigation of the moment when the dominance occurs, due to substantial cognitive demands of the task. Moreover, it indicates a need for deeper understanding of how cognitive load affects the physiological arousal effect, in an attempt to inform the design of such tasks. Furthermore, just like in the effects observed in the pupil, an increased task difficulty increases the heart rate (HR) [7, 39]. So the same suggestion for the pupil is applicable to the heart, where it is unclear to which extent is the cognitive activity uniquely associated with the physiological arousal response .
This investigation is motivated by these findings, as it follows up on the previous study  to include also an on–line biofeedback presentation, where participants have been engaged in a decision-making task. The identification of cognitive load through the pupil is already recognized in the community . The authors argue that traditional physiological methods of detecting arousal need to be enhanced with cognitive load of individuals, as the cognitive load can affect physiological changes and mask the arousal information in data. Moreover, it can have a negative effect on players’ emotions, which might disrupt the acquisition of the intended skill. Recent findings in the serious game design take into consideration an individual participant information to provide an enhanced experience , moving towards player-centered design . Due to the differences in personality and skill, diverse players would have different performance on the task in serious games. For some players whom the challenge is too high, this would either reduce performance on the task, push them towards an unwanted emotional state, or disrupt the acquisition of the skill they are trying to master. Depending on the design goals of serious games, this might be unacceptable. Therefore, it may be beneficial to provide a feedback for the participants to shape awareness of their own cognitive abilities and to tailor serious games to the cognitive abilities of an individual player. Usage of unobtrusive on–line physiological signals is necessary to asses the substantial cognitive load of players engaged on a decision–making task in serious games. Since pupillary response will show an arousal effect only when the cognitive demands of a situation are minimal, these modalities may provide the information of the moment when the dominance of the cognitive load over the physiological arousal effect occurs. Players might be engaged in a challenging situation at such moment, where game designers might take action to mitigate the disruptive effects of such states and change the difficulty of a task for each individual player to achieve the design goals of serious games. Therefore, the research question this study focuses on is to find the moment when substantial cognitive load overshadows the arousal effect, in an attempt to investigate cognitive abilities of the participants engaged on a decision-making task. Following those propositions, an infrared eye-tracking system and an electrocardiograph (ECG) sensor emerged as a solution providing quantitative measurements of the moment when substantial cognitive load overshadows the arousal effect . Therefore, a dynamic serious game environment with an on–line biofeedback has been used. These insights could provide a deeper understanding of how physiological arousal is underlying cognitive load in such decision-making tasks from the perspective of a human participant, potentially informing the design of more meaningful serious game tasks that would be tailored to the cognitive abilities of an individual interacting with them.
The remainder of the paper is structured as follows: background is given in Section 2, Section 3 presents the experimental set-up and methodological approach. Results are given in Section 4, discussion and limitations are given in Section 5 and Section 6 respectively. Conclusions are detailed in Section 7.
2.1 Serious games
Serious games can be defined as games with a purpose other than just entertainment, where they are aiming at captivating and engaging a participant for a specific purpose [25, 43]. They are rewarding investigation platforms since participants must explore their options, even though complete or clear information about the best course of action is not available . Serious games support higher motivation on the task, as it is significantly correlated with the players’ ability to handle cognitive load . The authors state that the individuals with higher motivation can handle a task using lower cognitive load processing ability. Physiology integrates continuous and unobtrusive measures of physiological arousal in serious games . Such physiology metrics offer insights into the participant’s experience , and behavior [16, 38]. Among a variety of eye-tracking techniques, pupil diameter is already a familiar metric for the experience and behavior in serious games, due to its ecological validity and ease of use .
2.2 Eye pupil and responses
Thus, Stanners et al.  concludes that the pupil response will show an arousal effect only when the cognitive demands of the situation are minimal. If the situation requires a substantial level of cognitive activity, only that activity will be observable on the pupil diameter, dominating over the physiological arousal effect indicated by the pupillary response. In short, minimal cognitive load results in the physiological arousal effect present in the pupil response. Substantial cognitive load results in the pupil response overshadowing the physiological arousal effect. Steinhauer et al.  gave neuroscientific support for this theory where they report that multiple pathways impinge on the Edinger–Westphal complex, resulting in pupillary dilation through inhibition of the parasympathetic pathway. Those multiple pathways that generally increase in inhibition across all task conditions may well include contributions of both direct cortical/indirect cortico–thalamic-hypothalamic pathways  and reticular pathways contributing to arousal . It has been shown that the pupil size reflects processing load or mental effort [29, 36]. Thus, Steinhauer et al.  state that demanding cognitive load, most likely associated with frontal cortical functioning, contributes heavily to this inhibitory process dominating over the arousal effect.
Granholm et al.  state that the changes in pupillary motility have been noted and employed as indicators of emotional arousal. A proposition came from Partala et al.  to use pupil size variation as a computer input signal in affective computing. Bradley et al.  present a strong case supporting that the pupil’s response during affective picture viewing reflects emotional arousal associated with the increased sympathetic activity. They argue that emotional arousal is a key element in modulating the pupil’s response. Furthermore, they state that rather than varying with cardiac deceleration mediated by the differences in parasympathetic activity, they have found that pupillary changes during emotionally arousing picture viewing predominantly reflects sympathetic nervous system activity. While in the previous studies exploring the effects of mental load and sustained cognitive processing on the pupil, the observed pupillary dilation appears to be mediated by parasympathetic inhibition of the sphincter muscle. These findings support the hypothesis that pupillary changes during the affective picture viewing are mediated by increased sympathetic activity and strongly suggest that emotional arousal affects the pupil diameter, independent of whether the pictures are pleasant or unpleasant in valence. These insights support the suggestion  that it is unclear to which extent is the cognitive activity uniquely associated with sympathetic or parasympathetic nervous system activity. The authors further argue that, if we are to provide quantitative measurements of activation underlying the central nervous system mechanisms, it is important to differentiate between activation of these pathways. With that knowledge, one can evaluate the neurophysiological systems that contribute to cognitive activities by monitoring pupillary dynamics.
2.3 Heart and physiological arousal
Berntson et al.  state that HR is a measure of both the sympathetic and parasympathetic autonomic nervous system activity. Correspondingly, sympathetic activity tends to increase HR, while parasympathetic activity decreases it. This suggests that this sympathetically mediated response covaries with emotional arousal . In contrast, cardiac deceleration is generally greater when viewing unpleasant, compared to either pleasant or neutral pictures . This suggests a parasympathetic activity , as shown in the pharmacological blockade studies of fear bradycardia in animals. The term bradycardia is defined as HR which falls below 60 beats per minute (BPM). Similarly to the pupil, Berntson et al.  further report that the extent of interplay between sympathetic or parasympathetic nervous system activity provides a measure of a subject’s emotion–regulation capabilities. Studies have found a strong correlation between the HR and physiological arousal [3, 44]. Just like the effects observed in the pupil, it has been found that increased task difficulty (e.g., difficulty of a mental arithmetic task) increases HR [7, 39]. ECG has been successfully used as a measure for assessing physiological emotions in humans with more than 80 % success rate [2, 35]. The method of biofeedback using HR became ubiquitous for targeting towards enhancing individuals’ abilities .
This investigation presents a dynamic serious game environment with an on–line biofeedback, where participants have been engaged in a decision-making task. Using such serious game together with an infrared eye-tracking system and an ECG sensor allowed for the assessment of physiological arousal and underlying the cognitive load in players engaged on the decision–making task. Therefore, the research question this study focuses on is to find the moment when substantial cognitive load overshadows the arousal effect, in an attempt to investigate cognitive abilities of the participants engaged on a decision-making task in serious games. These insights could provide a deeper understanding of how physiological arousal is underlying cognitive load in such decision-making tasks, providing a feedback for the players to shape awareness of their own cognitive abilities and tailor serious games to it.
A regression relationship study was conducted where the participants interacted with a serious game while the continuous measurements of HR and PD were made.
Twenty-one students participated in the experiment. The age range of the participants was between 20-24 years. Participants have not reported any ophthalmological problems (other than corrected vision), also no psychiatric or major medical disorders. Participants were given a movie ticket as a reward for participating. All participants received complete information on the study’s goal, experimental conditions and gave their informed consent.
3.2 Experimental setup
Participants were seated in a recliner chair in a small, sound-attenuated room. Lighting and temperature were controlled in such a way that artificial fixture light was used throughout the experiment, while the temperature was held constant at 23 °C ± 1 °C. The chest band holding a physiological Movisens ekgMove ECG sensor was attached. Participants were seated in front of and facing the Tobii T60 eye-tracker at a 50-60 cm distance from the screen.
Physiological signals were acquired using the Tobii T60 Eye-tracker for the PD data, with data acquisition frequency of 60 Hz. In addition, ECG was acquired at the chest using the Movisens ekgMove sensor, which records signals with high accuracy, while still providing a less obtrusive measurement to most other commercially available ECG devices. It uses flexible belt with dry electrodes, with two contact electrodes between the left and the right lowest rib points. The raw ECG signal is transmitted via Bluetooth to the algorithm module processing the current arousal level of the players from the HR information . The arousal levels are computed in relation to a baseline period recorded before the game starts, and transferred to the Auction Game. The PD was measured using an eye-tracker, whose data were analyzed offline. Computation of the HR from the ECG data provided information about the current physiological arousal level of the participants. Those arousal levels were computed against the baseline period, which was recorded for five minutes in a resting state before the game starts.
3.3 Experimental task
The decision-making task was presented in the following steps, shown in Fig. 3. Prior to the onset of price estimations, the participants had one second pause. Every price-estimation signal was displayed on the screen for one second, as a rule of thumb in fast–paced decision-making. The participants then had two seconds to reach a buy or sell decision, and shorter as the trials progressed or physiological arousal increased. After a decision, the feedback conveying the outcome of a decision was presented. The participant traded fictional ’goods’ and only the correct price was relevant for the participant. For each traded good the participants received three price estimations indicated on the screen. The mean value of these price estimations was the correct price for reaching a buy or sell decision for an offered price. Depending on a participant’s decision, s/he could have earned or lost virtual money (on average 2.50 €) relating to a correct or incorrect decision. The decision had to be made within the allocated amount of time, otherwise a participant was punished with a loss (-5 €). Such player–centered serious game design using biofeedback where physiological arousal was linked to the time and difficulty of the decision-making task, allowed each individual participant to regulate their emotions and perform their maximum on the decision–making task reducing the influence of arousal. This allowed the observation of the participants’ performance on the decision–making task limited by his/her ability to handle the cognitive load, reducing the influence of physiological arousal.
Prior to the experimental session, a calibration of the eye-tracker was performed where the participants had to focus attentively on four individual focus points distributed randomly on the screen, after which the participants were instructed to start with the experiment. The luminance of the screen and the light conditions around the screen were measured and controlled for all trials.
Eye-tracking baseline recording of the individual PD variations was obtained from the three still color images (white, gray and black) presented for 60 seconds each, counterbalanced between the participants in order to minimize the ordering effect.
Practice run of the serious game was rehearsed in order to acquaint the participants with the task.
Arousal baseline for the HR measurements was taken in the resting phase prior to the game’s onset while the fixation cross was presented in the middle of the screen. Participants were directed to relax for five minutes while the HR baseline recording was acquired.
The serious game was executed as described in the Experimental Setup section, presented in the Fig. 3.
Following the experiment, the ECG sensor was removed. The experimenter subsequently debriefed and thanked the participant. The participants were given a movie ticket for their participation.
3.5 Data analysis
Offline analysis was performed to obtain the PD values in relation to the HR values, to quantify physiological arousal data. In the offline analysis, HR data were used purely as a measure of the task difficulty.
3.5.1 Heart rate
The raw ECG signal was amplified, filtered using a band-pass filter at 10–40 Hz, and 16–bit digitized, the signal was then smoothed by a 10 ms moving average window. The R-peak (heartbeat) detection was based on OSEA algorithm  and it consisted of identification of R-waves applied to the ECG signal to obtain the interbeat intervals which were then reduced to HR, measured in BPM. The peaks were then detected in the resulting signal and detection heuristic rules were applied to avoid missing R-peaks or detecting multiple peaks for a single heartbeat. Artifacts were removed following the recommendations by , where a minimum interbeat interval of 300 ms and maximum interval of 1500 ms between peaks were enforced, constraining the interval to be 20% different from the previous one. Furthermore, the peaks were checked for both positive and negative slopes to ensure that the baseline drift is not misclassified as a peak, while the detected missing peaks were backtracked with reexamination/interpolation. Such interbeat intervals were analyzed for ectopic beats (disturbance in the cardiac rhythm), arrhythmic events, missing data and noise effects outliers, which were linearly interpolated.
3.5.2 Pupil diameter
Regarding pupil data, off-line analysis of individual trials was carried out for the PD values in millimeters. The data were corrected for the short and long blink periods. A linear interpolation was then applied to the short blink periods. Baseline diameter was defined as the average diameter during the 60 s still image presentation (i.e., white, gray, and black screens), which was used to normalize the pupil diameter data used in the subsequent PD analyses. The PD maxima, minima and the mean absolute values for each eye were averaged across all the participants for the baseline measurements of each pupil and each participant, indicating where the center values are. The order of the presentations was counterbalanced between participants in order to minimize ordering effects.
Cognitive load and physiological arousal on a decision-making task were investigated to find the moment when one prevails over the other. To perform this investigation, the PD data were analyzed in relation to the HR data during this challenging task. A statistical post–hoc power analysis was performed for that relationship using the sample data from the experiment. The effect size in this study was medium (N = 1499, f2 = .264), considering Cohen’s criteria . With an alpha level of p < .05, the statistical power for this study exceeded .99 for detecting a medium effect. Data were analyzed for non–linear regression in the SPSS statistical software package to estimate the relationship between these data variables. The data showed no violation of normality or homoscedasticity (data were normalized and the artifacts were removed before the analysis).
Data showed the maximum PD occurring across the HR values for the decision-making task. This coupling allowed for the observation of a substantial cognitive load dominating over the arousal effect on PD, and their dominance over each other, as determined by a non–linear regression on the values of PD data in relation to the HR data for the overall number (n = 1499) of measurements in the whole experiment. The data had a quadratic component that can be described by a best-fit curve having a non-zero slope. The regression analysis calculated the equation of the best-fit curve as y = − 2.713 + 0.128x − 0.0006x2 with max(y) = 4.11 mm occuring at x = 106.67BPM(n = 1499, p = .003, R2 = .209). The best–fit curve showed the moment when the dominance occurs with the PD maximum of 4.11 mm occurring for the HR of 106.67 BPM. It has been overlaid over the data averages across all participants and trials, shown on Fig. 4 in red and black respectively. Moreover, the Fig. 4 shows an interesting relationship between PD to HR, which greatly resembles the Yerkes-Dodson reversed ’U’ curve .
The intention of this research was to investigate how substantial cognitive load overshadows the physiological arousal on the PD data , in a serious game task. More specifically, the investigation was focused on the behavior of the PD in relation to the HR signal. This investigation was based on a challenging decision-making task in a serious game with on-line biofeedback of physiological arousal.
Findings from this experiment suggest that the PD shows the moment when a substantial cognitive load overshadows the physiological arousal effect in relation to HR data, shown in Fig. 4. This effect increased the PD with physiological arousal until the moment when the dominance occurs, as supported by the previous findings [40, 41]. The mapping of PD to cognitive load, while arousal was mapped to the HR axis, was motivated in the context of this study. This indicated the empirical model of Yerkes-Dodson according to which performance increases with arousal up to a critical arousal point, only to decrease passed that point . Under the assumption of higher performance during periods of increased cognitive load, an optimal level of physiological arousal results in maximum performance on a given task. Taking findings from this experiment together with the previous investigations claiming that a challenging task increases HR [7, 39] and PD [29, 36], it can be suggested that the PD could be a useful measure of cognitive abilities of participants. However, coupled with another measure (e.g., HR), because of the non-monotonous behavior of the PD across the increasing physiological arousal values. These findings may indicate a potential of serious games to apply physiological metrics to identifying the moment when a substantial cognitive load overshadows the arousal effect, to provide game tasks tailed to the cognitive abilities of an individual interacting with them . These findings were also supported through the previous research, where it was found that the pupillary response will show an arousal effect only when the cognitive demands of a situation are minimal [40, 41].
Following each presentation of the price estimation stimulus on the screen, there was a possibility of a pupil diameter change, as well as constriction due to the light reflex effect. Averaging the data over the experiment should have removed these artifacts. Moreover, the fast-paced and highly arousing context of The Auction Game does not allow for the investigation of each individual event in a trial, but an average across a single trial, due to the slow-response HR changes. There might have been a natural physiological delay in the pupillary response followed by the heart response. Therefore, it made sense to average the PD over the period of a heartbeat. The context of the experiment does not allow for the analysis of more short-term measures, due to the nature of the physiological modalities involved, mainly the HR. Therefore, this study focused mainly on the relationship between the PD and HR physiological signals, to establish the possibility of assessing cognitive load from the data. Therefore, the focus wan not on the effect of correct/incorrect decision-making. Furthermore, the serious game context of this experiment might have limited the generalizability of the results towards any decision-making task. Future studies would need to investigate other decision-making task contexts.
This research contributes to the current body of knowledge by having used the serious game displaying on–line biofeedback of physiological arousal to the participants. Moreover, it successfully identified the moment when a substantial cognitive load overshadowed the arousal effect. These results indicate that the coupling between PD and HR as a measure of physiological arousal may be a useful indicator of cognitive abilities of the participants in the context of a challenging decision-making task. As the task becomes challenging, requiring a substantial cognitive load, dominance of the cognitive load over the physiological arousal effect was observed on the PD. Moreover, HR continued to increase due to the physiological arousal and task difficulty, as supported by previous research . These findings could provide a step towards understanding the activation underlying the central nervous system mechanisms, and their contribution to cognitive activities. Moreover, it may be beneficial to provide a feedback for the participants to shape awareness of their own cognitive abilities. Usage of physiological signals and observations of the moment when dominance of the cognitive load over the physiological arousal effect occurs could make possible the tailoring of serious games to the cognitive abilities of an individual participant, as previously suggested .
In summary, this study focused on the research question to find the moment when substantial cognitive load overshadows the arousal effect, in an attempt to investigate cognitive abilities of the participants engaged on a decision-making task. The key contributions of this paper include: identification of the moment when the participants’ cognitive abilities are not sufficient to perform on the difficult task in the serious game; furthermore, the activation of PNS and SNS have been mapped where their dominance over each–other has been observed, during the course of the task; moreover, the optimal arousal has been identified where maximal performance and the cognitive load have been utilized on the task in the serious game, which is motivated by the reversed ’U’ curve relationship between arousal an performance in the Yerkes-Dodson effect .
Future studies are therefore necessary to investigate how a dynamically tailored serious game would influence the participants’ cognitive abilities and performance on a decision-making task.
- 3.Andreassi JL (2013) Psychophysiology: Human behavior and physiological response, 5th edn. Psychology PressGoogle Scholar
- 6.Berntson GG, Quigley KS, Lozano D (2007) Cardiovascular psychophysiology. In: Cacioppo J T, Tassinary L G, Berntson G G (eds) Handbook of psychophysiology. Cambridge Univ. Press, Cambridge, pp 182–210Google Scholar
- 10.Charles D, Black M (2004) Dynamic player modelling: a framework for player-centred digital games. In: Proceedings of 5th international conference on computer games: artificial intelligence, design and education (CGAIDE’04), pp 29–35Google Scholar
- 11.Charles D, Kerr A, McNeill M (2005) Player-centred game design: Player modelling and adaptive digital games. Digital Games 285(6):285–298. https://doi.org/10.1.1.97.735Google Scholar
- 12.Clifford G (2007) ECG Statistics, noise, artifacts, and missing data. In: Advanced methods and tools for ECG data analysis, chap 3. Artech House, Boston, pp 55–100Google Scholar
- 13.Cohen J (1988) Statistical power analysis for the behavioral sciencesGoogle Scholar
- 14.Cohen RA (2011) Yerkes–Dodson law. In: Encyclopedia of clinical neuropsychology. Springer, pp 2737–2738Google Scholar
- 16.Guardini P, Maninetti P (2013) Better game experience through game metrics: a rally videogame case study. In: Game analytics. Springer, pp 325–361Google Scholar
- 21.Jerčiċ P, Astor PJ, Adam M, Hilborn O, Schaff K, Lindley CA, Sennersten C, Eriksson J (2012) A serious game using physiological interfaces for emotion regulation training in the context of financial decision-making. In: ECIS 2012 proceedings, AIS electronic library (AISeL), p 2012Google Scholar
- 25.Léger PM, Charland P, Feldstein HD, Robert J, Babin G, Lyle D (2011) Business simulation training in information technology education: guidelines for new approaches in IT training. J Inf Technol Educ 10(1):39–53Google Scholar
- 26.Lindley C, Nacke L, Sennersten C (2008) Dissecting play-investigating the cognitive and emotional motivations and affects of computer gameplay. In: International conference on computer games (CGames), Citeseer, vol 13, pp 3–5Google Scholar
- 28.McAllister G, Mirza-Babaei P, Avent J (2013) Improving gameplay with game metrics and player metrics. In: Game analytics. Springer, pp 621–638Google Scholar
- 30.Nacke LE, Drachen A, Kuikkaniemi K, Kort YaWD (2009) Playability and player experience research. In: Proceedings of DiGRA, pp 1–11. https://doi.org/10.1145/1324198.1324208
- 37.Schaaff K, Mueller L, Kirst M, Heuer S (2011) Improving financial decision making using psychophysiological sensor data. In: 51st Annual Meeting of Society for Psychophysiological ResGoogle Scholar
- 38.Sennersten CC, Lindley CA (2009) An investigation of visual attention in FPS computer gameplay. In: Conference in Games and virtual worlds for serious applications, 2009. VS-GAMES’09. IEEE, pp 68–75Google Scholar
- 43.Susi T, Johannesson M (2007) Serious games: an overview. Tech. rep., Institutionen for kommunikation och informationGoogle Scholar
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.