Skip to main content
SpringerLink
Log in
Book cover

International Conference on Human-Computer Interaction

HCII 2020: Engineering Psychology and Cognitive Ergonomics. Mental Workload, Human Physiology, and Human Energy pp 3–13Cite as

Towards a Multimodal Measure for Physiological Behaviours to Estimate Cognitive Load

Part of the Lecture Notes in Computer Science book series (LNAI,volume 12186)

Abstract

We present an experiment investigating the relationships between different physiological measures, including Mean Pupil Diameter Change, Blinking-Rate, Heart-Rate, and Heart-Rate Variability to inform the development of a measure to estimate Cognitive Load. Our experiment involved participants performing a task to spot correct or incorrect words and sentences which successfully induced Cognitive Load. Our results show that participants’ task performance predicts their subjective rating of Cognitive Load and that there was a decrease in participants’ performance with an increase in Cognitive Load. Furthermore, Mean Pupil Diameter Change was able to predict Blinking-Rate, and Heart-Rate was able to predict Heart-Rate Variability. This prediction is evidence that collecting data on physiological behaviours synchronously and analysing the trends can be an effective way of estimating Cognitive Load, and will help the future development of an online measure of Cognitive Load useful for responsive user interfaces.

Keywords

  • Cognitive load
  • Mental load
  • Human-computer interaction
  • Pupillometry

This is a preview of subscription content, access via your institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   39.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   54.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

Notes

  1. 1.

    https://github.com/BrutusTT/tobii_api.

  2. 2.

    https://github.com/BrutusTT/ml_study.

  3. 3.

    H1a1: Participants’ subjective ratings of CL will predict their overall task performances (H1a1).

  4. 4.

    H1b: Lower participants’ ratings of CL will predict better task performance or vice versa.

  5. 5.

    H1a2: Participants’ subjective ratings of CL will predict time spent to complete the tasks.

  6. 6.

    H2: Participants’ changes in one physiological behaviour will predict a change in another behaviour.

  7. 7.

    H2b: Changes in the overall mean value of HR will predict overall mean HRV and vice versa. Similarly, BR will predict overall mean PD and vice versa.

References

  1. Corsense elite HRV device. https://elitehrv.com/corsense

  2. Psychopy. https://www.psychopy.org

  3. Tobii eye-tracking glasses. https://www.tobiipro.com/product-listing/tobii-pro-glasses-2/

  4. Ahmad, M., Keller, I., Lohan, K.S.: Integrated real-time, non-intrusive measurements for mental load. In: CHI 2019 Workshop: Everyday Automation Experience: Non-Expert Users Encountering Ubiquitous Automated Systems (2019)

    Google Scholar 

  5. Berthold, A., Jameson, A.: Interpreting symptoms of cognitive load in speech input. In: Kay, J. (ed.) UM99 User Modeling. CICMS, vol. 407, pp. 235–244. Springer, Vienna (1999). https://doi.org/10.1007/978-3-7091-2490-1_23

    CrossRef  Google Scholar 

  6. Consortium, B., et al.: The British national corpus, version 3 (BNC XML edition) (2007). Distributed by Oxford University Computing Services on behalf of the BNC Consortium. http://www.natcorp.ox.ac.uk. Accessed 25 May 2012

  7. Craig, C.L., et al.: International physical activity questionnaire: 12-country reliability and validity. Med. Sci. Sports Exerc. 35(8), 1381–1395 (2003)

    CrossRef  Google Scholar 

  8. Cranford, K.N., Tiettmeyer, J.M., Chuprinko, B.C., Jordan, S., Grove, N.P.: Measuring load on working memory: the use of heart rate as a means of measuring chemistry students’ cognitive load. J. Chem. Educ. 91(5), 641–647 (2014)

    CrossRef  Google Scholar 

  9. Fogelholm, M., et al.: International physical activity questionnaire: validity against fitness. Med. Sci. Sports Exerc. 38(4), 753–760 (2006)

    CrossRef  Google Scholar 

  10. Gregoire, J., Tuck, S., Hughson, R.L., Yamamoto, Y.: Heart rate variability at rest and exercise: influence of age, gender, and physical training. Can. J. Appl. Physiol. 21(6), 455–470 (1996)

    CrossRef  Google Scholar 

  11. Haapalainen, E., Kim, S., Forlizzi, J.F., Dey, A.K.: Psycho-physiological measures for assessing cognitive load. In: Proceedings of the 12th ACM International Conference on Ubiquitous Computing, pp. 301–310. ACM (2010)

    Google Scholar 

  12. Hart, S.G.: NASA-task load index (NASA-TLX); 20 years later. In: Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 50, pp. 904–908. Sage Publications, Los Angeles (2006)

    Google Scholar 

  13. Holland, M.K., Tarlow, G.: Blinking and mental load. Psychol. Rep. 31(1), 119–127 (1972)

    CrossRef  Google Scholar 

  14. Hrv, E.: How do you calculate the HRV score? Webpage, July 2018. https://help.elitehrv.com/article/54-how-do-you-calculate-the-hrv-score

  15. Hrv, E.: Corsense heart rate variability. Webpage, January 2019. https://elitehrv.com/corsense

  16. Jameson, A., Kiefer, J., Müller, C., Großmann-Hutter, B., Wittig, F., Rummer, R.: Assessment of a user’s time pressure and cognitive load on the basis of features of speech. In: Crocker, M., Siekmann, J. (eds.) Resource-Adaptive Cognitive Processes, pp. 171–204. Springer, Heidelberg (2010). https://doi.org/10.1007/978-3-540-89408-7_9

  17. Khawaja, M.A., Ruiz, N., Chen, F.: Think before you talk: an empirical study of relationship between speech pauses and cognitive load. In: Proceedings of the 20th Australasian Conference on Computer-Human Interaction: Designing for Habitus and Habitat, pp. 335–338. ACM (2008)

    Google Scholar 

  18. Kret, M.E., Sjak-Shie, E.E.: Preprocessing pupil size data: guidelines and code. Behav. Res. Methods 51(3), 1336–1342 (2018). https://doi.org/10.3758/s13428-018-1075-y

    CrossRef  Google Scholar 

  19. Mukherjee, S., Yadav, R., Yung, I., Zajdel, D.P., Oken, B.S.: Sensitivity to mental effort and test-retest reliability of heart rate variability measures in healthy seniors. Clin. Neurophysiol. 122(10), 2059–2066 (2011)

    Google Scholar 

  20. Noel, J.B., Bauer Jr., K.W., Lanning, J.W.: Improving pilot mental workload classification through feature exploitation and combination: a feasibility study. Comput. Oper. Res. 32(10), 2713–2730 (2005)

    CrossRef  Google Scholar 

  21. Paas, F., Tuovinen, J.E., Tabbers, H., Van Gerven, P.W.: Cognitive load measurement as a means to advance cognitive load theory. Educ. Psychol. 38(1), 63–71 (2003)

    CrossRef  Google Scholar 

  22. Palinko, O., Kun, A.L., Shyrokov, A., Heeman, P.: Estimating cognitive load using remote eye tracking in a driving simulator. In: Proceedings of the 2010 Symposium on Eye-Tracking Research & Applications, pp. 141–144. ACM (2010)

    Google Scholar 

  23. Pang, B., Lee, L.: A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In: Proceedings of the ACL (2004)

    Google Scholar 

  24. Reilly, J., Kelly, A., Kim, S.H., Jett, S., Zuckerman, B.: The human task-evoked pupillary response function is linear: Implications for baseline response scaling in pupillometry. Behav. Res. Methods 51(2), 865–878 (2018). https://doi.org/10.3758/s13428-018-1134-4

    CrossRef  Google Scholar 

  25. Sabyruly, Y., Broz, F., Keller, I., Lohan, K.S.: Gaze and attention during an HRI storytelling task. In: 2015 AAAI Fall Symposium Series (2015)

    Google Scholar 

  26. Sweller, J.: Cognitive load theory, learning difficulty, and instructional design. Learn. Instr. 4(4), 295–312 (1994)

    CrossRef  Google Scholar 

  27. Sweller, J., Van Merrienboer, J.J., Paas, F.G.: Cognitive architecture and instructional design. Educ. Psychol. Rev. 10(3), 251–296 (1998)

    CrossRef  Google Scholar 

  28. Van Gog, T., Kester, L., Paas, F.: Effects of concurrent monitoring on cognitive load and performance as a function of task complexity. Appl. Cogn. Psychol. 25(4), 584–587 (2011)

    CrossRef  Google Scholar 

  29. Wilson, G.F., Russell, C.A.: Real-time assessment of mental workload using psychophysiological measures and artificial neural networks. Hum. Factors 45(4), 635–644 (2003)

    CrossRef  Google Scholar 

  30. Yin, B., Chen, F., Ruiz, N., Ambikairajah, E.: Speech-based cognitive load monitoring system. In: 2008 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 2041–2044. IEEE (2008)

    Google Scholar 

  31. Zhang, J., Yin, Z., Wang, R.: Recognition of mental workload levels under complex human-machine collaboration by using physiological features and adaptive support vector machines. IEEE Trans. Hum.-Mach. Syst. 45(2), 200–214 (2015)

    CrossRef  Google Scholar 

Download references

Acknowledgment

The authors would like to acknowledge the support of the ORCA Hub EPSRC (EP/R026173/1, 2017-2021) and consortium partners.

Author information

Authors and Affiliations

  1. Edinburgh Center for Robotics, Computer Science Department, Heriot-Watt University, Edinburgh, UK

    Muneeb Imtiaz Ahmad, David A. Robb, Ingo Keller & Katrin Lohan

  2. NTB University of Applied Sciences in Technology, Buchs, Switzerland

    Katrin Lohan

Authors
  1. Muneeb Imtiaz Ahmad
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. David A. Robb
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ingo Keller
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Katrin Lohan
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Muneeb Imtiaz Ahmad , David A. Robb , Ingo Keller or Katrin Lohan .

Editor information

Editors and Affiliations

  1. Coventry University, Coventry, UK

    Prof. Don Harris

  2. Cranfield University, Cranfield, UK

    Prof. Wen-Chin Li

Rights and permissions

Reprints and Permissions

Copyright information

© 2020 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Ahmad, M.I., Robb, D.A., Keller, I., Lohan, K. (2020). Towards a Multimodal Measure for Physiological Behaviours to Estimate Cognitive Load. In: Harris, D., Li, WC. (eds) Engineering Psychology and Cognitive Ergonomics. Mental Workload, Human Physiology, and Human Energy. HCII 2020. Lecture Notes in Computer Science(), vol 12186. Springer, Cham. https://doi.org/10.1007/978-3-030-49044-7_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-49044-7_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-49043-0

  • Online ISBN: 978-3-030-49044-7

  • eBook Packages: Computer ScienceComputer Science (R0)

search

Navigation