Modeling cognitive load and physiological arousal through pupil diameter and heart rate

  • Petar Jerčić
  • Charlotte Sennersten
  • Craig Lindley
Open Access


This study investigates individuals’ cognitive load processing abilities while engaged on a decision-making task in serious games, to explore how a substantial cognitive load dominates over the physiological arousal effect on pupil diameter. A serious game was presented to the participants, which displayed the on–line biofeedback based on physiological measurements of arousal. In such dynamic decision-making environment, the pupil diameter was analyzed in relation to the heart rate, to evaluate if the former could be a useful measure of cognitive abilities of individuals. As pupil might reflect both cognitive activity and physiological arousal, the pupillary response will show an arousal effect only when the cognitive demands of the situation are minimal. Evidence shows that in a situation where a substantial level of cognitive activity is required, only that activity will be observable on the pupil diameter, dominating over the physiological arousal effect indicated by the pupillary response. It is suggested that it might be possible to design serious games tailored to the cognitive abilities of an individual player, using the proposed physiological measurements to observe the moment when such dominance occurs.


Serious games Physiology Electrocardiogram Pupil diameter Arousal Cognitive load 

1 Introduction

Research shows that emotions impair or facilitate decision-making [1, 19]. Russell’s model [34] classifies emotions through valence and arousal, their independent components. Therefore, valence represents whether the current emotional experience of a situation is positive or negative. On another hand, the level of excitement is represented by arousal. Following this classification, emotions are measured through the valence and arousal components. The concept of physiological arousal has been generally validated in models of emotion [33]. Task difficulty is correlated with physiological arousal [14], thus a challenging decision task might be reflected in physiological signals. The author further states that there is a moment when performance decreases as arousal continues to increase. The pupil dilation is influenced by physiological arousal, together with attention and interest [3, 5]. The pupil diameter (PD) has been validated as a useful measure of physiological arousal [15, 23]. Early investigators [18] viewed the pupillary response as reflecting the level of arousal or emotionality; followed a few years later [18, 22] by studies arguing that the pupillary response might reflect cognitive activity as well as or instead of arousal. Thus, Stanners et al. [40] concludes that the pupil response will show an arousal effect only when the cognitive demands of the situation are minimal. If the situation requires a substantial level of cognitive activity, only that activity will be observable on the pupil diameter, dominating over the physiological arousal effect indicated by the pupillary response [32]. These findings give rise to a need for the investigation of the moment when the dominance occurs, due to substantial cognitive demands of the task. Moreover, it indicates a need for deeper understanding of how cognitive load affects the physiological arousal effect, in an attempt to inform the design of such tasks. Furthermore, just like in the effects observed in the pupil, an increased task difficulty increases the heart rate (HR) [7, 39]. So the same suggestion for the pupil is applicable to the heart, where it is unclear to which extent is the cognitive activity uniquely associated with the physiological arousal response [41].

This investigation is motivated by these findings, as it follows up on the previous study [41] to include also an on–line biofeedback presentation, where participants have been engaged in a decision-making task. The identification of cognitive load through the pupil is already recognized in the community [20]. The authors argue that traditional physiological methods of detecting arousal need to be enhanced with cognitive load of individuals, as the cognitive load can affect physiological changes and mask the arousal information in data. Moreover, it can have a negative effect on players’ emotions, which might disrupt the acquisition of the intended skill. Recent findings in the serious game design take into consideration an individual participant information to provide an enhanced experience [10], moving towards player-centered design [11]. Due to the differences in personality and skill, diverse players would have different performance on the task in serious games. For some players whom the challenge is too high, this would either reduce performance on the task, push them towards an unwanted emotional state, or disrupt the acquisition of the skill they are trying to master. Depending on the design goals of serious games, this might be unacceptable. Therefore, it may be beneficial to provide a feedback for the participants to shape awareness of their own cognitive abilities and to tailor serious games to the cognitive abilities of an individual player. Usage of unobtrusive on–line physiological signals is necessary to asses the substantial cognitive load of players engaged on a decision–making task in serious games. Since pupillary response will show an arousal effect only when the cognitive demands of a situation are minimal, these modalities may provide the information of the moment when the dominance of the cognitive load over the physiological arousal effect occurs. Players might be engaged in a challenging situation at such moment, where game designers might take action to mitigate the disruptive effects of such states and change the difficulty of a task for each individual player to achieve the design goals of serious games. Therefore, the research question this study focuses on is to find the moment when substantial cognitive load overshadows the arousal effect, in an attempt to investigate cognitive abilities of the participants engaged on a decision-making task. Following those propositions, an infrared eye-tracking system and an electrocardiograph (ECG) sensor emerged as a solution providing quantitative measurements of the moment when substantial cognitive load overshadows the arousal effect [41]. Therefore, a dynamic serious game environment with an on–line biofeedback has been used. These insights could provide a deeper understanding of how physiological arousal is underlying cognitive load in such decision-making tasks from the perspective of a human participant, potentially informing the design of more meaningful serious game tasks that would be tailored to the cognitive abilities of an individual interacting with them.

The remainder of the paper is structured as follows: background is given in Section 2, Section 3 presents the experimental set-up and methodological approach. Results are given in Section 4, discussion and limitations are given in Section 5 and Section 6 respectively. Conclusions are detailed in Section 7.

2 Background

2.1 Serious games

Serious games can be defined as games with a purpose other than just entertainment, where they are aiming at captivating and engaging a participant for a specific purpose [25, 43]. They are rewarding investigation platforms since participants must explore their options, even though complete or clear information about the best course of action is not available [25]. Serious games support higher motivation on the task, as it is significantly correlated with the players’ ability to handle cognitive load [42]. The authors state that the individuals with higher motivation can handle a task using lower cognitive load processing ability. Physiology integrates continuous and unobtrusive measures of physiological arousal in serious games [30]. Such physiology metrics offer insights into the participant’s experience [28], and behavior [16, 38]. Among a variety of eye-tracking techniques, pupil diameter is already a familiar metric for the experience and behavior in serious games, due to its ecological validity and ease of use [26].

2.2 Eye pupil and responses

The pupil is an opening at the center of the iris surrounded by the iris muscles that respond to outside stimuli to control the amount of light that reaches the retina. We differentiate between two types of iris muscles: the sphincter muscle sphincter pupillae forming a band around the inner margin of the pupil and radial muscle dilator pupillae, see Fig. 1. Two synergistic pathways control the pupil size and dynamics. Those pathways are the parasympathetic pathway and the sympathetic pathway, operating on the smooth muscles of the pupil. The parasympathetic pathway is mediated with the efferent pathway originating in the Edinger–Westphal oculomotor complex in the midbrain and innervates the sphincter, which is the circular muscle responsible for the constriction, as seen in the reflex reaction to light. Inhibition of Edinger–Westphal complex results in relaxation of the sphincter muscles and, thus, significant dilation. The sympathetic pathway, mediated by the hypothalamus, innervates the radial dilator muscle of the iris responsible for dilation [41].
Fig. 1

Model of the autonomic nervous system control pathways for the pupil diameter and the HR

Thus, Stanners et al. [40] concludes that the pupil response will show an arousal effect only when the cognitive demands of the situation are minimal. If the situation requires a substantial level of cognitive activity, only that activity will be observable on the pupil diameter, dominating over the physiological arousal effect indicated by the pupillary response. In short, minimal cognitive load results in the physiological arousal effect present in the pupil response. Substantial cognitive load results in the pupil response overshadowing the physiological arousal effect. Steinhauer et al. [41] gave neuroscientific support for this theory where they report that multiple pathways impinge on the Edinger–Westphal complex, resulting in pupillary dilation through inhibition of the parasympathetic pathway. Those multiple pathways that generally increase in inhibition across all task conditions may well include contributions of both direct cortical/indirect cortico–thalamic-hypothalamic pathways [27] and reticular pathways contributing to arousal [41]. It has been shown that the pupil size reflects processing load or mental effort [29, 36]. Thus, Steinhauer et al. [41] state that demanding cognitive load, most likely associated with frontal cortical functioning, contributes heavily to this inhibitory process dominating over the arousal effect.

Granholm et al. [15] state that the changes in pupillary motility have been noted and employed as indicators of emotional arousal. A proposition came from Partala et al. [31] to use pupil size variation as a computer input signal in affective computing. Bradley et al. [9] present a strong case supporting that the pupil’s response during affective picture viewing reflects emotional arousal associated with the increased sympathetic activity. They argue that emotional arousal is a key element in modulating the pupil’s response. Furthermore, they state that rather than varying with cardiac deceleration mediated by the differences in parasympathetic activity, they have found that pupillary changes during emotionally arousing picture viewing predominantly reflects sympathetic nervous system activity. While in the previous studies exploring the effects of mental load and sustained cognitive processing on the pupil, the observed pupillary dilation appears to be mediated by parasympathetic inhibition of the sphincter muscle. These findings support the hypothesis that pupillary changes during the affective picture viewing are mediated by increased sympathetic activity and strongly suggest that emotional arousal affects the pupil diameter, independent of whether the pictures are pleasant or unpleasant in valence. These insights support the suggestion [41] that it is unclear to which extent is the cognitive activity uniquely associated with sympathetic or parasympathetic nervous system activity. The authors further argue that, if we are to provide quantitative measurements of activation underlying the central nervous system mechanisms, it is important to differentiate between activation of these pathways. With that knowledge, one can evaluate the neurophysiological systems that contribute to cognitive activities by monitoring pupillary dynamics.

2.3 Heart and physiological arousal

Berntson et al. [6] state that HR is a measure of both the sympathetic and parasympathetic autonomic nervous system activity. Correspondingly, sympathetic activity tends to increase HR, while parasympathetic activity decreases it. This suggests that this sympathetically mediated response covaries with emotional arousal [24]. In contrast, cardiac deceleration is generally greater when viewing unpleasant, compared to either pleasant or neutral pictures [8]. This suggests a parasympathetic activity [6], as shown in the pharmacological blockade studies of fear bradycardia in animals. The term bradycardia is defined as HR which falls below 60 beats per minute (BPM). Similarly to the pupil, Berntson et al. [6] further report that the extent of interplay between sympathetic or parasympathetic nervous system activity provides a measure of a subject’s emotion–regulation capabilities. Studies have found a strong correlation between the HR and physiological arousal [3, 44]. Just like the effects observed in the pupil, it has been found that increased task difficulty (e.g., difficulty of a mental arithmetic task) increases HR [7, 39]. ECG has been successfully used as a measure for assessing physiological emotions in humans with more than 80 % success rate [2, 35]. The method of biofeedback using HR became ubiquitous for targeting towards enhancing individuals’ abilities [2].

This investigation presents a dynamic serious game environment with an on–line biofeedback, where participants have been engaged in a decision-making task. Using such serious game together with an infrared eye-tracking system and an ECG sensor allowed for the assessment of physiological arousal and underlying the cognitive load in players engaged on the decision–making task. Therefore, the research question this study focuses on is to find the moment when substantial cognitive load overshadows the arousal effect, in an attempt to investigate cognitive abilities of the participants engaged on a decision-making task in serious games. These insights could provide a deeper understanding of how physiological arousal is underlying cognitive load in such decision-making tasks, providing a feedback for the players to shape awareness of their own cognitive abilities and tailor serious games to it.

3 Methodology

A regression relationship study was conducted where the participants interacted with a serious game while the continuous measurements of HR and PD were made.

3.1 Participants

Twenty-one students participated in the experiment. The age range of the participants was between 20-24 years. Participants have not reported any ophthalmological problems (other than corrected vision), also no psychiatric or major medical disorders. Participants were given a movie ticket as a reward for participating. All participants received complete information on the study’s goal, experimental conditions and gave their informed consent.

3.2 Experimental setup

Participants were seated in a recliner chair in a small, sound-attenuated room. Lighting and temperature were controlled in such a way that artificial fixture light was used throughout the experiment, while the temperature was held constant at 23 °C ± 1 °C. The chest band holding a physiological Movisens ekgMove ECG sensor was attached. Participants were seated in front of and facing the Tobii T60 eye-tracker at a 50-60 cm distance from the screen.

Physiological signals were acquired using the Tobii T60 Eye-tracker for the PD data, with data acquisition frequency of 60 Hz. In addition, ECG was acquired at the chest using the Movisens ekgMove sensor, which records signals with high accuracy, while still providing a less obtrusive measurement to most other commercially available ECG devices. It uses flexible belt with dry electrodes, with two contact electrodes between the left and the right lowest rib points. The raw ECG signal is transmitted via Bluetooth to the algorithm module processing the current arousal level of the players from the HR information [37]. The arousal levels are computed in relation to a baseline period recorded before the game starts, and transferred to the Auction Game. The PD was measured using an eye-tracker, whose data were analyzed offline. Computation of the HR from the ECG data provided information about the current physiological arousal level of the participants. Those arousal levels were computed against the baseline period, which was recorded for five minutes in a resting state before the game starts.

Following the definition of serious games as games with a clear purpose other than just entertainment [43], The Auction Game [21] was used as the serious game in this study to investigate cognitive abilities of the participants and the moment when a substantial cognitive load overshadows the physiological arousal effect. The Auction Game has emerged as an investigation platform since it presents a challenging decision-making task to the participants, interconnected with the ECG sensor for the on–line assessment of physiological arousal and the biofeedback presentation based on HR values. An advantage of such coupling between the Auction Game and the physiological ECG sensor was that it measured reliable physiological arousal in a stressful environment [4, 21, 37]. The physiological arousal continuously adjusted the decision-making task and thereby making it more (or less) difficult, depending on the participant’s ability to regulate physiological arousal. The participants were given information regarding their physiological arousal through the meter in the upper right corner and through the color of the clouds, see Fig. 2. Furthermore, they were informed that regulating their arousal would benefit them on the task presented.
Fig. 2

The Auction Game – a serious game used in this study presents a challenging decision-making task to participants, directly linked to their physiological arousal state indicated via the biofeedback [21]. The biofeedback information regarding physiological arousal was given through the meter in the upper right corner and through the color of the clouds

3.3 Experimental task

The participants were presented with a decision-making task to calculate a mean value from three given price estimations, to be able to reach a buy or sell decision at the correct price, see Fig. 3. To make a decision, a participant had to click on the buy or sell button on the screen. Price estimations were directly linked to the physiological arousal level, such that they deviated from the correct price with higher variance the more aroused the participant was. Thus, a lower physiological arousal would make the variance of price estimations closer to the correct price, so that a buy or sell decision could have been easier. The serious game was linear, which meant that there was always just one possible correct decision to be made in each trial, and it was a reasonable challenge for most of the participants. Good emotion–regulation skills would benefit the participants to make the decision-making task less challenging, as higher physiological arousal also reduced the decision time, to promote a greater challenge. Moreover, the task became more challenging at subsequent trials as the decision period was further reduced, forcing a quick decision. On average, subjects performed 71 trials before the game stopped, with 19 trials being the fewest and 160 being the most.
Fig. 3

A trial consisted of three parts: (1) a rest period, where the participant is allowed to rest before the stimulus onset (0 – 1 s); (2) a stimulus onset period, where the participant is shown the three price estimations from which a correct price should be calculated that they should use to make a decision (1 – 4 s); and (3) an decision period, where the participant is making a buy/sell decision–based on the correct price and the offer given (4 – 6 s). The length of the decision period was dependent on the participant’s level of arousal and getting even shorter as the trials progressed, minimum being one second

The decision-making task was presented in the following steps, shown in Fig. 3. Prior to the onset of price estimations, the participants had one second pause. Every price-estimation signal was displayed on the screen for one second, as a rule of thumb in fast–paced decision-making. The participants then had two seconds to reach a buy or sell decision, and shorter as the trials progressed or physiological arousal increased. After a decision, the feedback conveying the outcome of a decision was presented. The participant traded fictional ’goods’ and only the correct price was relevant for the participant. For each traded good the participants received three price estimations indicated on the screen. The mean value of these price estimations was the correct price for reaching a buy or sell decision for an offered price. Depending on a participant’s decision, s/he could have earned or lost virtual money (on average 2.50 €) relating to a correct or incorrect decision. The decision had to be made within the allocated amount of time, otherwise a participant was punished with a loss (-5 €). Such player–centered serious game design using biofeedback where physiological arousal was linked to the time and difficulty of the decision-making task, allowed each individual participant to regulate their emotions and perform their maximum on the decision–making task reducing the influence of arousal. This allowed the observation of the participants’ performance on the decision–making task limited by his/her ability to handle the cognitive load, reducing the influence of physiological arousal.

3.4 Protocol

Upon arrival, the participants were given general information about the experiment and the description explaining the serious game. Their written consent was obtained, explaining physiological measurements and data confidentiality. When the participants agreed to take part in the study, they signed an informed consent form. Each experimental session took around 20 minutes to complete.
  1. 1.

    Prior to the experimental session, a calibration of the eye-tracker was performed where the participants had to focus attentively on four individual focus points distributed randomly on the screen, after which the participants were instructed to start with the experiment. The luminance of the screen and the light conditions around the screen were measured and controlled for all trials.

  2. 2.

    Eye-tracking baseline recording of the individual PD variations was obtained from the three still color images (white, gray and black) presented for 60 seconds each, counterbalanced between the participants in order to minimize the ordering effect.

  3. 3.

    Practice run of the serious game was rehearsed in order to acquaint the participants with the task.

  4. 4.

    Arousal baseline for the HR measurements was taken in the resting phase prior to the game’s onset while the fixation cross was presented in the middle of the screen. Participants were directed to relax for five minutes while the HR baseline recording was acquired.

  5. 5.

    The serious game was executed as described in the Experimental Setup section, presented in the Fig. 3.

  6. 6.

    Following the experiment, the ECG sensor was removed. The experimenter subsequently debriefed and thanked the participant. The participants were given a movie ticket for their participation.


3.5 Data analysis

Offline analysis was performed to obtain the PD values in relation to the HR values, to quantify physiological arousal data. In the offline analysis, HR data were used purely as a measure of the task difficulty.

3.5.1 Heart rate

The raw ECG signal was amplified, filtered using a band-pass filter at 10–40 Hz, and 16–bit digitized, the signal was then smoothed by a 10 ms moving average window. The R-peak (heartbeat) detection was based on OSEA algorithm [17] and it consisted of identification of R-waves applied to the ECG signal to obtain the interbeat intervals which were then reduced to HR, measured in BPM. The peaks were then detected in the resulting signal and detection heuristic rules were applied to avoid missing R-peaks or detecting multiple peaks for a single heartbeat. Artifacts were removed following the recommendations by [12], where a minimum interbeat interval of 300 ms and maximum interval of 1500 ms between peaks were enforced, constraining the interval to be 20% different from the previous one. Furthermore, the peaks were checked for both positive and negative slopes to ensure that the baseline drift is not misclassified as a peak, while the detected missing peaks were backtracked with reexamination/interpolation. Such interbeat intervals were analyzed for ectopic beats (disturbance in the cardiac rhythm), arrhythmic events, missing data and noise effects outliers, which were linearly interpolated.

3.5.2 Pupil diameter

Regarding pupil data, off-line analysis of individual trials was carried out for the PD values in millimeters. The data were corrected for the short and long blink periods. A linear interpolation was then applied to the short blink periods. Baseline diameter was defined as the average diameter during the 60 s still image presentation (i.e., white, gray, and black screens), which was used to normalize the pupil diameter data used in the subsequent PD analyses. The PD maxima, minima and the mean absolute values for each eye were averaged across all the participants for the baseline measurements of each pupil and each participant, indicating where the center values are. The order of the presentations was counterbalanced between participants in order to minimize ordering effects.

Normalized PD values were grouped according to the accompanying HR values recorded. The normalized PD values were then averaged over HR values and plotted as the graph, shown on Fig. 4. The PD values were normalized for each participant using the BLmin = 1.34 mm and BLmax = 6.15 mm obtained from the baseline period, using the normalizing (1).
$$ PD_{norm} = \left( \frac{DP_{t}-DP_{min}}{DP_{max}-DP_{min}}\right)\times\left( BL_{max}-BL_{min} \right)+BL_{min} $$
Fig. 4

The normalized pupil diameter responses in relation to the HR on a decision-making task in a serious game. The substantial cognitive load dominating over the physiological arousal effect is observable on the PD as the HR increases. The black line represents the averaged PD values in relation to HR across participants and trials in the experiment. Red line represents best-fit non-linear quadratic regression curve for the PD values in relation to HR for all participants and trials over the whole experiment with 95% confidence interval at the .003 probability level

4 Results

Cognitive load and physiological arousal on a decision-making task were investigated to find the moment when one prevails over the other. To perform this investigation, the PD data were analyzed in relation to the HR data during this challenging task. A statistical post–hoc power analysis was performed for that relationship using the sample data from the experiment. The effect size in this study was medium (N = 1499, f2 = .264), considering Cohen’s criteria [13]. With an alpha level of p < .05, the statistical power for this study exceeded .99 for detecting a medium effect. Data were analyzed for non–linear regression in the SPSS statistical software package to estimate the relationship between these data variables. The data showed no violation of normality or homoscedasticity (data were normalized and the artifacts were removed before the analysis).

Data showed the maximum PD occurring across the HR values for the decision-making task. This coupling allowed for the observation of a substantial cognitive load dominating over the arousal effect on PD, and their dominance over each other, as determined by a non–linear regression on the values of PD data in relation to the HR data for the overall number (n = 1499) of measurements in the whole experiment. The data had a quadratic component that can be described by a best-fit curve having a non-zero slope. The regression analysis calculated the equation of the best-fit curve as y = − 2.713 + 0.128x − 0.0006x2 with max(y) = 4.11 mm occuring at x = 106.67BPM(n = 1499, p = .003, R2 = .209). The best–fit curve showed the moment when the dominance occurs with the PD maximum of 4.11 mm occurring for the HR of 106.67 BPM. It has been overlaid over the data averages across all participants and trials, shown on Fig. 4 in red and black respectively. Moreover, the Fig. 4 shows an interesting relationship between PD to HR, which greatly resembles the Yerkes-Dodson reversed ’U’ curve [14].

5 Discussion

The intention of this research was to investigate how substantial cognitive load overshadows the physiological arousal on the PD data [41], in a serious game task. More specifically, the investigation was focused on the behavior of the PD in relation to the HR signal. This investigation was based on a challenging decision-making task in a serious game with on-line biofeedback of physiological arousal.

Findings from this experiment suggest that the PD shows the moment when a substantial cognitive load overshadows the physiological arousal effect in relation to HR data, shown in Fig. 4. This effect increased the PD with physiological arousal until the moment when the dominance occurs, as supported by the previous findings [40, 41]. The mapping of PD to cognitive load, while arousal was mapped to the HR axis, was motivated in the context of this study. This indicated the empirical model of Yerkes-Dodson according to which performance increases with arousal up to a critical arousal point, only to decrease passed that point [14]. Under the assumption of higher performance during periods of increased cognitive load, an optimal level of physiological arousal results in maximum performance on a given task. Taking findings from this experiment together with the previous investigations claiming that a challenging task increases HR [7, 39] and PD [29, 36], it can be suggested that the PD could be a useful measure of cognitive abilities of participants. However, coupled with another measure (e.g., HR), because of the non-monotonous behavior of the PD across the increasing physiological arousal values. These findings may indicate a potential of serious games to apply physiological metrics to identifying the moment when a substantial cognitive load overshadows the arousal effect, to provide game tasks tailed to the cognitive abilities of an individual interacting with them [7]. These findings were also supported through the previous research, where it was found that the pupillary response will show an arousal effect only when the cognitive demands of a situation are minimal [40, 41].

6 Limitations

Following each presentation of the price estimation stimulus on the screen, there was a possibility of a pupil diameter change, as well as constriction due to the light reflex effect. Averaging the data over the experiment should have removed these artifacts. Moreover, the fast-paced and highly arousing context of The Auction Game does not allow for the investigation of each individual event in a trial, but an average across a single trial, due to the slow-response HR changes. There might have been a natural physiological delay in the pupillary response followed by the heart response. Therefore, it made sense to average the PD over the period of a heartbeat. The context of the experiment does not allow for the analysis of more short-term measures, due to the nature of the physiological modalities involved, mainly the HR. Therefore, this study focused mainly on the relationship between the PD and HR physiological signals, to establish the possibility of assessing cognitive load from the data. Therefore, the focus wan not on the effect of correct/incorrect decision-making. Furthermore, the serious game context of this experiment might have limited the generalizability of the results towards any decision-making task. Future studies would need to investigate other decision-making task contexts.

7 Conclusions

This research contributes to the current body of knowledge by having used the serious game displaying on–line biofeedback of physiological arousal to the participants. Moreover, it successfully identified the moment when a substantial cognitive load overshadowed the arousal effect. These results indicate that the coupling between PD and HR as a measure of physiological arousal may be a useful indicator of cognitive abilities of the participants in the context of a challenging decision-making task. As the task becomes challenging, requiring a substantial cognitive load, dominance of the cognitive load over the physiological arousal effect was observed on the PD. Moreover, HR continued to increase due to the physiological arousal and task difficulty, as supported by previous research [41]. These findings could provide a step towards understanding the activation underlying the central nervous system mechanisms, and their contribution to cognitive activities. Moreover, it may be beneficial to provide a feedback for the participants to shape awareness of their own cognitive abilities. Usage of physiological signals and observations of the moment when dominance of the cognitive load over the physiological arousal effect occurs could make possible the tailoring of serious games to the cognitive abilities of an individual participant, as previously suggested [11].

In summary, this study focused on the research question to find the moment when substantial cognitive load overshadows the arousal effect, in an attempt to investigate cognitive abilities of the participants engaged on a decision-making task. The key contributions of this paper include: identification of the moment when the participants’ cognitive abilities are not sufficient to perform on the difficult task in the serious game; furthermore, the activation of PNS and SNS have been mapped where their dominance over each–other has been observed, during the course of the task; moreover, the optimal arousal has been identified where maximal performance and the cognitive load have been utilized on the task in the serious game, which is motivated by the reversed ’U’ curve relationship between arousal an performance in the Yerkes-Dodson effect [14].

Future studies are therefore necessary to investigate how a dynamically tailored serious game would influence the participants’ cognitive abilities and performance on a decision-making task.



  1. 1.
    Adam MTP, Krämer J, Müller MB (2015) Auction fever! how time pressure and social competition affect bidders’ arousal and bids in retail auctions. J Retail 91(3):468–485. CrossRefGoogle Scholar
  2. 2.
    Al Osman H, Eid M, El Saddik A (2014) U-biofeedback: a multimedia-based reference model for ubiquitous biofeedback systems. Multimedia Tools and Applications 72(3):3143–3168. CrossRefGoogle Scholar
  3. 3.
    Andreassi JL (2013) Psychophysiology: Human behavior and physiological response, 5th edn. Psychology PressGoogle Scholar
  4. 4.
    Astor PJ, Adam MTP, Jerčić P, Schaff K, Weinhardt C (2014) Integrating biosignals into information systems: a neurois tool for improving emotion regulation. J Manag Inf Syst 30(3):247–278CrossRefGoogle Scholar
  5. 5.
    Benikos N, Johnstone SJ, Roodenrys SJ (2013) Varying task difficulty in the Go/Nogo task: the effects of inhibitory control, arousal, and perceived effort on ERP components. Int J Psychophysiol 87(3):262–272. 10.1016/j.ijpsycho.2012.08.005CrossRefGoogle Scholar
  6. 6.
    Berntson GG, Quigley KS, Lozano D (2007) Cardiovascular psychophysiology. In: Cacioppo J T, Tassinary L G, Berntson G G (eds) Handbook of psychophysiology. Cambridge Univ. Press, Cambridge, pp 182–210Google Scholar
  7. 7.
    Boutcher YN, Boutcher SH (2006) Cardiovascular response to stroop: effect of verbal response and task difficulty. Biol Psychol 73(3):235–241. CrossRefGoogle Scholar
  8. 8.
    Bradley MM, Codispoti M, Cuthbert BN, Lang PJ (2001) Emotion and motivation I: defensive and appetitive reactions in picture processing. Emotion 1 (3):276–298. CrossRefGoogle Scholar
  9. 9.
    Bradley MM, Miccoli L, Ma E, Lang PJ (2008) The pupil as a measure of emotional arousal and autonomic activation. Psychophysiology 45(4):602–7. CrossRefGoogle Scholar
  10. 10.
    Charles D, Black M (2004) Dynamic player modelling: a framework for player-centred digital games. In: Proceedings of 5th international conference on computer games: artificial intelligence, design and education (CGAIDE’04), pp 29–35Google Scholar
  11. 11.
    Charles D, Kerr A, McNeill M (2005) Player-centred game design: Player modelling and adaptive digital games. Digital Games 285(6):285–298. Scholar
  12. 12.
    Clifford G (2007) ECG Statistics, noise, artifacts, and missing data. In: Advanced methods and tools for ECG data analysis, chap 3. Artech House, Boston, pp 55–100Google Scholar
  13. 13.
    Cohen J (1988) Statistical power analysis for the behavioral sciencesGoogle Scholar
  14. 14.
    Cohen RA (2011) Yerkes–Dodson law. In: Encyclopedia of clinical neuropsychology. Springer, pp 2737–2738Google Scholar
  15. 15.
    Granholm E, Steinhauer SR (2004) Pupillometric measures of cognitive and emotional processes. Int J Psychophysiol 52(1):1–6. CrossRefGoogle Scholar
  16. 16.
    Guardini P, Maninetti P (2013) Better game experience through game metrics: a rally videogame case study. In: Game analytics. Springer, pp 325–361Google Scholar
  17. 17.
    Hamilton P (2002) Open source ECG analysis. Comput Cardiol 29:101–104CrossRefGoogle Scholar
  18. 18.
    Hess EH, Polt JM (1964) Pupil size in relation to mental activity during simple problem-solving. Science 143(3611):1190–1192. CrossRefGoogle Scholar
  19. 19.
    Hu Y, Wang D, Pang K, Xu G, Guo J (2015) The effect of emotion and time pressure on risk. Journal of Risk Research 18(5):637–650. CrossRefGoogle Scholar
  20. 20.
    Hung JCS, Chiang KH, Huang YH, Lin KC (2017) Augmenting teacher-student interaction in digital learning through affective computing. Multimedia Tools and Applications 76(18):18,361–18,386. CrossRefGoogle Scholar
  21. 21.
    Jerčiċ P, Astor PJ, Adam M, Hilborn O, Schaff K, Lindley CA, Sennersten C, Eriksson J (2012) A serious game using physiological interfaces for emotion regulation training in the context of financial decision-making. In: ECIS 2012 proceedings, AIS electronic library (AISeL), p 2012Google Scholar
  22. 22.
    Kahneman D, Beatty J (1966) Pupil diameter and load on memory. Science 145:1583–1585CrossRefGoogle Scholar
  23. 23.
    Laeng B, Sirois S, Gredeback G (2012) Pupillometry: a window to the preconscious? Perspect Psychol Sci 7(1):18–27. CrossRefGoogle Scholar
  24. 24.
    Lang PJ, Greenwald MK, Bradley MM, Hamm AO (1993) Looking at pictures: affective, facial, vis- ceral, and behavioral reactions. Psychophysiology 30:261–273CrossRefGoogle Scholar
  25. 25.
    Léger PM, Charland P, Feldstein HD, Robert J, Babin G, Lyle D (2011) Business simulation training in information technology education: guidelines for new approaches in IT training. J Inf Technol Educ 10(1):39–53Google Scholar
  26. 26.
    Lindley C, Nacke L, Sennersten C (2008) Dissecting play-investigating the cognitive and emotional motivations and affects of computer gameplay. In: International conference on computer games (CGames), Citeseer, vol 13, pp 3–5Google Scholar
  27. 27.
    Lowenstein O (1955) Pupillary reflex shapes and topical clinical diagnosis. BMC Neurology 5(9):631–. CrossRefGoogle Scholar
  28. 28.
    McAllister G, Mirza-Babaei P, Avent J (2013) Improving gameplay with game metrics and player metrics. In: Game analytics. Springer, pp 621–638Google Scholar
  29. 29.
    Moresi S, Adam JJ, Rijcken J, Van Gerven PWM, Kuipers H, Jolles J (2008) Pupil dilation in response preparation. Int J Psychophysiol 67(2):124–30. CrossRefGoogle Scholar
  30. 30.
    Nacke LE, Drachen A, Kuikkaniemi K, Kort YaWD (2009) Playability and player experience research. In: Proceedings of DiGRA, pp 1–11.
  31. 31.
    Partala T, Surakka V (2003) Pupil size variation as an indication of affective processing. International Journal of Human-Computer Studies 59(1–2):185–198. CrossRefGoogle Scholar
  32. 32.
    Pehlivanoglu D, Jain S, Ariel R, Verhaeghen P (2014) The ties to unbind: Age-related differences in feature (un)binding in working memory for emotional faces. Front Psychol 5:1–13. CrossRefGoogle Scholar
  33. 33.
    Porges SW (2007) The polyvagal perspective. Biol Psychol 74(2):116–143., NIHMS150003CrossRefGoogle Scholar
  34. 34.
    Posner J, Ja R, Peterson BS (2005) The circumplex model of affect: An integrative approach to affective neuroscience, cognitive development, and psychopathology. Dev Psychopathol 17 (3):715–734., NIHMS150003CrossRefGoogle Scholar
  35. 35.
    Rani P, Sarkar N, Adams J (2007) Anxiety-based affective communication for implicit human–machine interaction. Adv Eng Inform 21(3):323–334CrossRefGoogle Scholar
  36. 36.
    Recarte MA, Nunes L (2003) Mental workload while driving: Effects on visual search, discrimination, and decision making. J Exp Psychol Appl 9(2):119–133CrossRefGoogle Scholar
  37. 37.
    Schaaff K, Mueller L, Kirst M, Heuer S (2011) Improving financial decision making using psychophysiological sensor data. In: 51st Annual Meeting of Society for Psychophysiological ResGoogle Scholar
  38. 38.
    Sennersten CC, Lindley CA (2009) An investigation of visual attention in FPS computer gameplay. In: Conference in Games and virtual worlds for serious applications, 2009. VS-GAMES’09. IEEE, pp 68–75Google Scholar
  39. 39.
    Sosnowski T, Krzywosz-Rynkiewicz B, Roguska J (2004) Program running versus problem solving: mental task effect on tonic heart rate. Psychophysiology 41 (3):467–75. CrossRefGoogle Scholar
  40. 40.
    Stanners R, Coulter M, Sweet A (1979) The pupillary response as an indicator of arousal and cognition. Motiv Emot 3(4):319–340CrossRefGoogle Scholar
  41. 41.
    Steinhauer SR, Siegle GJ, Condray R, Pless M (2004) Sympathetic and parasympathetic innervation of pupillary dilation during sustained processing. Int J Psychophysiol 52(1):77–86. CrossRefGoogle Scholar
  42. 42.
    Su CH (2016) The effects of students’ motivation, cognitive load and learning anxiety in gamification software engineering education: a structural equation modeling study. Multimedia Tools and Applications 75(16):10,013–10,036. CrossRefGoogle Scholar
  43. 43.
    Susi T, Johannesson M (2007) Serious games: an overview. Tech. rep., Institutionen for kommunikation och informationGoogle Scholar
  44. 44.
    Xu Y, Liu G, Hao M, Wen W, Huang X (2010) Analysis of affective ECG signals toward emotion recognition. J Electron (China) 27(1):8–14CrossRefGoogle Scholar

Copyright information

© The Author(s) 2018

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Authors and Affiliations

  1. 1.Department of Creative TechnologiesBlekinge Institute of TechnologyKarlskronaSweden
  2. 2.CSIRO Mineral ResourcesTechnology CourtPullenvaleAustralia
  3. 3.Intelligent Sensing and Systems LaboratoryCSIRO ICT CentreHobartAustralia

Personalised recommendations