Multimodal Data Fusion for Person-Independent, Continuous Estimation of Pain Intensity

  • Markus Kächele
  • Patrick Thiam
  • Mohammadreza Amirian
  • Philipp Werner
  • Steffen Walter
  • Friedhelm Schwenker
  • Günther Palm
Conference paper
Part of the Communications in Computer and Information Science book series (CCIS, volume 517)


In this work, a method is presented for the continuous estimation of pain intensity based on fusion of bio-physiological and video features. The focus of the paper is to analyse which modalities and feature sets are suited best for the task of recognizing pain levels in a person-independent setting. A large set of features is extracted from the available bio-physiological channels (ECG, EMG and skin conductivity) and the video stream. Experimental validation demonstrates which modalities contribute the most to a robust prediction and the effects when combining them to improve the continuous estimation given unseen persons.


Root Mean Square Error Pain Intensity Local Binary Pattern Facial Region Continuous Estimation 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Artemiadis, P.K., Kyriakopoulos, K.J.: An EMG-based robot control scheme robust to time-varying EMG signal features. Trans. Info. Tech. Biomed. 14(3), 582–588 (2010)CrossRefGoogle Scholar
  2. 2.
    Ashraf, A.B., Lucey, S., Cohn, J.F., Chen, T., Ambadar, Z., Prkachin, K.M., Solomon, P.E.: The painful face - pain expression recognition using active appearance models. Image and Vision Computing 27(12), 1788–1796 (2009)CrossRefGoogle Scholar
  3. 3.
    Benedek, M., Kaernbach, C.: Decomposition of skin conductance data by means of nonnegative deconvolution. Psychophysiology 47(4), 647–658 (2010)Google Scholar
  4. 4.
    Cao, C., Slobounov, S.: Application of a novel measure of EEG non-stationarity as ’Shannon- entropy of the peak frequency shifting’ for detecting residual abnormalities in concussed individuals. Clinical neurophysiology : Official Journal of the International Federation of Clinical Neurophysiology 122(7), 1314–1321 (2011)CrossRefGoogle Scholar
  5. 5.
    Chen, W., Zhuang, J., Yu, W., Wang, Z.: Measuring complexity using FuzzyEn, ApEn, and SampEn. Medical Engineering & Physics 31(1), 61–68 (2009)CrossRefGoogle Scholar
  6. 6.
    Hammal, Z., Cohn, J.F.: Automatic detection of pain intensity. In: Proceedings of the International Conference on Multimodal Interaction, ICMI 2012, pp. 47–52. ACM (2012)Google Scholar
  7. 7.
    Kächele, M., Schels, M., Schwenker, F.: Inferring depression and affect from application dependent meta knowledge. In: Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge, AVEC 2014, pp. 41–48. ACM (2014)Google Scholar
  8. 8.
    Kächele, M., Werner, P., Al-Hamadi, A., Palm, G., Walter, S., Schwenker, F.: Bio-visual fusion for person-independent recognition of pain intensity. In: Schwenker, F., Roli, F., Kittler, J. (eds.) MCS 2015. LNCS, vol. 9132, pp. 220–230. Springer, Heidelberg (2015) CrossRefGoogle Scholar
  9. 9.
    Kaltwang, S., Rudovic, O., Pantic, M.: Continuous pain intensity estimation from facial expressions. In: Bebis, G., Boyle, R., Parvin, B., Koracin, D., Fowlkes, C., Wang, S., Choi, M.-H., Mantler, S., Schulze, J., Acevedo, D., Mueller, K., Papka, M. (eds.) ISVC 2012, Part II. LNCS, vol. 7432, pp. 368–377. Springer, Heidelberg (2012) CrossRefGoogle Scholar
  10. 10.
    Lucey, P., Cohn, J.F., Prkachin, K.M., Solomon, P.E., Matthews, I.: Painful data: The UNBC-McMaster shoulder pain expression archive database. Image, Vision, and Computing Journal, 197–205 (2012)Google Scholar
  11. 11.
    Phinyomark, A., Limsakul, C., Phukpattaranont, P.: A novel feature extraction for robust EMG pattern recognition. Journal of Computing 1(1), 71–80 (2009)Google Scholar
  12. 12.
    Schwenker, F., Dietrich, C.R., Thiel, C., Palm, G.: Learning of decision fusion mappings for pattern recognition. International Journal on Artificial Intelligence and Machine Learning (AIML) 6, 17–21 (2006)Google Scholar
  13. 13.
    Tkach, D., Huang, H., Kuiken, T.A.: Research study of stability of time-domain features for electromyographic pattern recognition. J. Neuroeng Rehabil 7, 21 (2010)CrossRefGoogle Scholar
  14. 14.
    Treister, R., Kliger, M., Zuckerman, G., Aryeh, I.G., Eisenberg, E.: Differentiating between heat pain intensities: The combined effect of multiple autonomic parameters. Pain 153(9), 1807–1814 (2012)CrossRefGoogle Scholar
  15. 15.
    Walter, S., Gruss, S., Ehleiter, H., Tan, J., Traue, H., Werner, P., Al-Hamadi, A., Crawcour, S., Andrade, A., Moreira da Silva, G.: The BioVid heat pain database data for the advancement and systematic validation of an automated pain recognition system. In: 2013 IEEE International Conference on Cybernetics (CYBCONF), pp. 128–131, June 2013Google Scholar
  16. 16.
    Walter, S., Gruss, S., Limbrecht-Ecklundt, K., Traue, H.C., Werner, P., Al-Hamadi, A., Diniz, N., Silva, GMd, Andrade, A.O.: Automatic pain quantification using autonomic parameters. Psychology & Neuroscience 7, 363–380 (2014)CrossRefGoogle Scholar
  17. 17.
    Werner, P., Al-Hamadi, A., Niese, R., Walter, S., Gruss, S., Traue, H.C.: Automatic pain recognition from video and biomedical signals. In: International Conference on Pattern Recognition, pp. 4582–4587 (2014)Google Scholar
  18. 18.
    Zhao, G., Pietikainen, M.: Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence 29(6), 915–928 (2007)CrossRefGoogle Scholar
  19. 19.
    Zhao, Q., Zhang, L.: ECG feature extraction and classification using wavelet transform and support vector machines. In: International Conference on Neural Networks and Brain, ICNN B 2005, vol. 2, pp. 1089–1092, October 2005Google Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Markus Kächele
    • 1
  • Patrick Thiam
    • 1
  • Mohammadreza Amirian
    • 1
  • Philipp Werner
    • 2
  • Steffen Walter
    • 3
  • Friedhelm Schwenker
    • 1
  • Günther Palm
    • 1
  1. 1.Institute of Neural Information ProcessingUlm UniversityUlmGermany
  2. 2.Institute of Information TechnologyUniversity of MagdeburgMagdeburgGermany
  3. 3.Department of Psychosomatic Medicine and PsychotherapyUlm UniversityUlmGermany

Personalised recommendations