Fusion Architectures for Multimodal Cognitive Load Recognition

  • Daniel Kindsvater
  • Sascha MeudtEmail author
  • Friedhelm Schwenker
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10183)


Knowledge about the users emotional state is important to achieve human like, natural Human Computer Interaction (HCI) in modern technical systems. Humans rely on implicit signals like body gestures and posture, vocal changes (e.g. pitch) and mimic expressions when communicating. We investigate the relation between them and human emotion, specifically when completing easy or difficult tasks. Additionally we include physiological data which also differ in changes of cognitive load. We focus on discriminating between mental overload and mental underload, which can e.g. be useful in an e-tutorial system. Mental underload is a new term used to describe the state a person is in when completing a dull or boring task. It will be shown how to select suited features, build uni modal classifiers which then are combined to a multimodal mental load estimation by the use of Markov Fusion Networks (MFN) and Kalman Filter Fusion (KFF).


Kalman Filter Cognitive Load Fusion Approach Classifier Decision Discrete Time Step 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.



The authors of this paper are partially funded by the Transregional Collaborative Research Centre SFB/TRR 62 “Companion-Technology for Cognitive Technical Systems” funded by the German Research Foundation (DFG).


  1. 1.
    Ekman, P.: An argument for basic emotions. Cogn. Emot. 6(3–4), 169–200 (1992)CrossRefGoogle Scholar
  2. 2.
    Glodek, M., Reuter, S., Schels, M., Dietmayer, K., Schwenker, F.: Kalman filter based classifier fusion for affective state recognition. In: Zhou, Z.H., Roli, F., Kittler, J. (eds.) MCS 2013. LNCS, vol. 7872, pp. 85–94. Springer, Heidelberg (2013). doi: 10.1007/978-3-642-38067-9_8 CrossRefGoogle Scholar
  3. 3.
    Glodek, M., Schels, M., Schwenker, F., Palm, G.: Combination of sequential class distributions from multiple channels using Markov fusion networks. J. Multimodal User Interfaces 8(3), 257–272 (2014)CrossRefGoogle Scholar
  4. 4.
    Held, D.: Bimodale Erkennung affektiver Zustnde durch Ensemble Methoden anhand von Audio- und Biosignalen. Master’s thesis, Ulm University (2016)Google Scholar
  5. 5.
    Hihn, H., Meudt, S., Schwenker, F.: Inferring mental overload based on postural behavior and gestures. In: Proceedings of the 2nd workshop on Emotion Representations and Modelling for Companion Systems. ACM (2016)Google Scholar
  6. 6.
    Kalman, R.E.: A new approach to linear filtering and prediction problems. J. Basic Eng. 82(1), 35–45 (1960)CrossRefGoogle Scholar
  7. 7.
    Kipp, M., Martin, J.-C.: Gesture and emotion: can basic gestural form features discriminate emotions? In: 3rd International Conference on Affective Computing and Intelligent Interaction Workshops, pp. 1–8. IEEE (2009)Google Scholar
  8. 8.
    Meudt, S., Zharkov, D., Kächele, M., Schwenker, F.: Multi classifier systems and forward backward feature selection algorithms to classify emotional coloured speech. In: Proceedings of the 15th ACM on International Conference on Multimodal Interaction, pp. 551–556. ACM (2013)Google Scholar
  9. 9.
    Picard, R.W.: Affective Computing, vol. 252. MIT Press, Cambridge (1997)Google Scholar
  10. 10.
    Plesa-Skwerer, D., Faja, S., Schofield, C., Verbalis, A., Tager-Flusberg, H., Dykens, E.M.: Perceiving facial and vocal expressions of emotion in individuals with Williams syndrome. Am. J. Ment. Retard. 111(1), 15–26 (2006)CrossRefGoogle Scholar
  11. 11.
    Russell, J.A., Bachorowski, J.-A., Fernández-Dols, J.-M.: Facial and vocal expressions of emotion. Annu. Rev. Psychol. 54(1), 329–349 (2003)CrossRefGoogle Scholar
  12. 12.
    Schels, M., Glodek, M., Meudt, S., Scherer, S., Schmidt, M., Layher, G., Tschechne, S., Brosch, T., Hrabal, D., Walter, S., et al. Multi-modal classifier-fusion for the recognition of emotions. In: Coverbal Synchrony in Human-Machine Interaction (2013)Google Scholar
  13. 13.
    Schüssel, F., Honold, F., Bubalo, N., Huckauf, A., Traue, H., Hazer-Rau, D.: In-depth analysis of multimodal interaction: an explorative paradigm. In: Kurosu, M. (ed.) HCI 2016. LNCS, vol. 9732, pp. 233–240. Springer, Cham (2016). doi: 10.1007/978-3-319-39516-6_22 CrossRefGoogle Scholar
  14. 14.
    Shan, C., Gong, S., McOwan, P.W.: Robust facial expression recognition using local binary patterns. In: IEEE International Conference on Image Processing, 2005. ICIP 2005, vol. 2, p. II-370. IEEE (2005)Google Scholar
  15. 15.
    Vogt, T., André, E., Bee, N.: EmoVoice — a framework for online recognition of emotions from voice. In: André, E., Dybkjær, L., Minker, W., Neumann, H., Pieraccini, R., Weber, M. (eds.) PIT 2008. LNCS, vol. 5078, pp. 188–199. Springer, Heidelberg (2008). doi: 10.1007/978-3-540-69369-7_21 CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Daniel Kindsvater
    • 1
  • Sascha Meudt
    • 1
    Email author
  • Friedhelm Schwenker
    • 1
  1. 1.Institute for Neural Information ProcessingUlm UniversityUlmGermany

Personalised recommendations