Evolving Systems

, Volume 8, Issue 1, pp 71–83 | Cite as

Adaptive confidence learning for the personalization of pain intensity estimation systems

  • Markus Kächele
  • Mohammadreza Amirian
  • Patrick Thiam
  • Philipp Werner
  • Steffen Walter
  • Günther Palm
  • Friedhelm Schwenker
Original Paper


In this work, a method is presented for the continuous estimation of pain intensity based on fusion of bio-physiological and video features. Furthermore, a method is proposed for the adaptation of the system to unknown test persons based on unlabeled data. First, an analysis is presented that shows which modalities and feature sets are suited best for the task of recognizing pain levels in a person-independent setting. For this, a large set of features is extracted from the available bio-physiological channels (ECG, EMG and skin conductivity) and the video stream. We then propose a method to learn the confidence of a regression system using a multi-stage ensemble classifier. Based on the outcome of the classifier, which is realized by a neural network, confident samples are selected by the adaptation procedure. In various experiments, we show that the algorithm is able to detect highly confident samples which can be used to improve the overall performance. We furthermore discuss the current limitations of automatic pain intensity estimation—in light of the presented approach and beyond.


Pain intensity estimation Multi-modal fusion Confidence estimation 


  1. Abdelwahab M, Busso C 2015) Supervised domain adaptation for emotion recognition speech. In: 2015 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2015, South Brisbane, Queensland, Australia, April 19–24, 2015, pp 5058–5062Google Scholar
  2. AlZoubi O, Fossati D, D’Mello S, Calvo RA (2014) Affect detection from non-stationary physiological data using ensemble classifiers. Evol Syst 6(2):79–92CrossRefGoogle Scholar
  3. AlZoubi O, Rafael C, Stevens R (2009) Classification of eeg for affect recognition: An adaptive approach. In: Nicholson A, Li X (eds) AI 2009: Advances in Artificial Intelligence, vol 5866., Lecture Notes in Computer ScienceSpringer, Berlin Heidelberg, pp 52–61Google Scholar
  4. Artemiadis PK, Kyriakopoulos KJ (2010) An EMG-based robot control scheme robust to time-varying EMG signal features. Trans Info Tech Biomed 14(3):582–588CrossRefGoogle Scholar
  5. Ashraf AB, Lucey S, Cohn JF, Chen T, Ambadar Z, Prkachin KM, Solomon PE (2009) The painful face—pain expression recognition using active appearance models. Image Vision Comput 27(12):1788–1796CrossRefGoogle Scholar
  6. Benedek M, Kaernbach C (2010) Decomposition of skin conductance data by means of nonnegative deconvolution. Psychophysiology 47(4):647–658Google Scholar
  7. Cao C, Slobounov S (2011) Application of a novel measure of EEG non-stationarity as ’Shannon- entropy of the peak frequency shifting’ for detecting residual abnormalities in concussed individuals. Clinical neurophysiology : Official Journal of the International Federation of Clinical Neurophysiology 122(7):1314–1321CrossRefGoogle Scholar
  8. Cauwenberghs G, Poggio T (2001) Incremental and decremental support vector machine learning. In: Advances in neural information processing systems, vol 13, pp 409–415Google Scholar
  9. Chen W, Zhuang J, Yu W, Wang Z (2009) Measuring complexity using FuzzyEn, ApEn, and SampEn. Medl Eng Phys 31(1):61–68CrossRefGoogle Scholar
  10. Duviella E, Serir L, Sayed-Mouchaweh M (2013) An evolving classification approach for fault diagnosis and prognosis of a wind farm. In: Conference on Control and Fault-Tolerant Systems (SysTol) IEEE, pp 377–382Google Scholar
  11. Glodek M, Tschechne S, Layher G, Schels M, Brosch T, Scherer S, Kächele M, Schmidt M, Neumann H, Palm G, Schwenker F (2011) Multiple classifier systems for the classification of audio-visual emotional states. In: D’Mello S, Graesser A, Schuller B, Martin JC (eds) Proceedings of the Fourth International Conference on Affective Computing and Intelligent Interaction (ACII’11) - Part II, LNCS 6975, Springer, pp 359–368Google Scholar
  12. Hammal Z, Cohn JF (2012) Automatic detection of pain intensity. In: Proceedings of the 14th ACM International Conference on Multimodal Interaction, ICMI ’12, ACM, pp 47–52Google Scholar
  13. He L, Jiang D, Yang L, Pei E, Wu P, Sahli H (2015) Multimodal affective dimension prediction using deep bidirectional long short-term memory recurrent neural networks. In: Proceedings of the 5th International Workshop on Audio/Visual Emotion Challenge, ACM , pp 73–80Google Scholar
  14. Kächele M, Glodek M, Zharkov D, Meudt S, Schwenker F (2014) Fusion of audio-visual features using hierarchical classifier systems for the recognition of affective states and the state of depression. In: M. De Marsico, A. Tabbone, A. Fred (eds.) Proceedings of the International Conference on Pattern Recognition Applications and Methods (ICPRAM), SciTePress, pp 671–678Google Scholar
  15. Kächele M, Schels M, Meudt S, Palm G, Schwenker F (2016) Revisiting the EmotiW challenge: how wild is it really? J Multimodal User Interfaces 10(2):151–162. doi: 10.1007/s12193-015-0202-7 CrossRefGoogle Scholar
  16. Kächele M, Schels M, Schwenker F (2014) Inferring depression and affect from application dependent meta knowledge. In: Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge, AVEC ’14, ACM, pp 41–48Google Scholar
  17. Kächele M, Thiam P, Amirian M, Schwenker F, Palm G (2016) Methods for person-centered continuous pain intensity assessment from bio-physiological channels. IEEE J Sel Topics Signal Proces (99), pp 1–1Google Scholar
  18. Kächele M, Thiam P, Amirian M, Werner P, Walter S, Schwenker F, Palm G (2015) Multimodal data fusion for person-independent, continuous estimation of pain intensity. In: Iliadis L, Jayne C (eds) Engineering Applications of Neural Networks, Communications in Computer and Information Science, vol. 517, pp. 275–285. Springer International PublishingGoogle Scholar
  19. Kächele M, Thiam P, Palm G, Schwenker F, Schels M (2015) Ensemble methods for continuous affect recognition: Multi-modality, temporality, and challenges. In: Proceedings of the 5th International Workshop on Audio/Visual Emotion Challenge, AVEC ’15, ACM, pp 9–16Google Scholar
  20. Kächele M, Werner P, Al-Hamadi A, Palm G, Walter S, Schwenker F (2015) Bio-visual fusion for person-independent recognition of pain intensity. In:  Schwenker F, Roli F, Kittler J (eds) Multiple classifier systems, Lecture notes in computer science, vol. 9132, pp 220–230. Springer International PublishingGoogle Scholar
  21. Kaltwang S, Rudovic O, Pantic M (2012) Continuous pain intensity estimation from facial expressions. In: Bebis G, Boyle R, Parvin B, Koracin D, Fowlkes C, Wang S, Choi MH, Mantler S, Schulze J, Acevedo D, Mueller K, Papka M (eds) Advances in visual computing, LNCS, vol 7432. Springer, Berlin, pp 368–377CrossRefGoogle Scholar
  22. Kuncheva L (2004) Combining pattern classifiers: methods and algorithms. Wiley, Hoboken, New JerseyGoogle Scholar
  23. Lucey P, Cohn JF, Prkachin KM, Solomon PE, Matthews I (2011) Painful data: the UNBC-McMaster shoulder pain expression archive database. In: IEEE International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011), IEEE, pp 57–64Google Scholar
  24. Ludl MC, Lewandowski A, Dorffner G (2008) Adaptive machine learning in delayed feedback domains by selective relearning. Appl Artif Intell 22(6):543–557CrossRefGoogle Scholar
  25. Phinyomark A, Limsakul C, Phukpattaranont P (2009) A novel feature extraction for robust EMG pattern recognition. J Comput 1(1):71–80Google Scholar
  26. Platt JC (1999) Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In: Advances in large margin classifiers, pp 61–74Google Scholar
  27. Schels M, Glodek M, Meud, S, Scherer S, Schmidt M, Layher G, Tschechne S, Brosch T, Hrabal D, Walter S, Traue H, Palm G, Neumann H, Schwenker F (2013) Multi-modal classifier-fusion for the recognition of emotions, 1 edn. Coverbal synchrony in human-machine interaction. CRC Press, USA, pp 73–97Google Scholar
  28. Schwenker F, Dietrich CR, Thiel C, Palm G (2006) Learning of decision fusion mappings for pattern recognition. Int J Artif Intell Mach Learn (AIML) 6:17–21Google Scholar
  29. Tkach D, Huang H, Kuiken TA (2010) Research study of stability of time-domain features for electromyographic pattern recognition. J Neuroeng Rehabil 7:21CrossRefGoogle Scholar
  30. Treister R, Kliger M, Zuckerman G, Aryeh IG, Eisenberg E (2012) Differentiating between heat pain intensities: the combined effect of multiple autonomic parameters. Pain 153(9):1807–1814CrossRefGoogle Scholar
  31. Walter S, Gruss S, Ehleiter H, Tan J, Traue H, Werner P, Al-Hamadi A, Crawcour S, Andrade A, Moreira da Silva G (2013) The BioVid heat pain database data for the advancement and systematic validation of an automated pain recognition system. In: IEEE International Conference on Cybernetics (CYBCONF), pp 128–131Google Scholar
  32. Walter S, Gruss S, Limbrecht-Ecklundt K, Traue HC, Werner P, Al-Hamadi A. Diniz N, Silva GMd, Andrade AO (2014) Automatic pain quantification using autonomic parameters. Psychol Neurosci 7:363–380CrossRefGoogle Scholar
  33. Wang W, Enescu V, Sahli H (2015) Adaptive real-time emotion recognition from body movements. ACM Trans Interactive Intell Syst 5(4):18:1–18:21Google Scholar
  34. Werner P, Al-Hamadi A, Niese R, Walter S, Gruss S, Traue HC (2014) Automatic pain recognition from video and biomedical signals. In: International Conference on Pattern Recognition, pp 4582–4587Google Scholar
  35. Wöllmer M, Kaiser M, Eyben F, Schuller B, Rigoll G (2013) LSTM-modeling of continuous emotions in an audiovisual affect recognition framework. Image Vision Comput 31(2):153–163CrossRefGoogle Scholar
  36. Yang J, Yan R, Hauptmann AG (2007) Cross-domain video concept detection using adaptive svms. In: Proceedings of the 15th ACM International Conference on Multimedia, MM ’07 , ACM pp 188–197Google Scholar
  37. Zhang C Hamid R, Zhang Z (2008) Taylor expansion based classifier adaptation: application to person detection. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, pp 1–8Google Scholar
  38. Zhao G, Pietikainen M (2007) Dynamic texture recognition using local binary patterns with an application to facial expressions. Pattern Analy Mach Intell IEEE Trans 29(6):915–928CrossRefGoogle Scholar
  39. Zhao Q, Zhang L (2005) ECG feature extraction and classification using wavelet transform and support vector machines. Int Conf Neural Netw Brain 2:1089–1092Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2016

Authors and Affiliations

  • Markus Kächele
    • 1
  • Mohammadreza Amirian
    • 1
  • Patrick Thiam
    • 1
  • Philipp Werner
    • 2
  • Steffen Walter
    • 3
  • Günther Palm
    • 1
  • Friedhelm Schwenker
    • 1
  1. 1.Institute of Neural Information Processing, Ulm UniversityUlmGermany
  2. 2.Institute of Information TechnologyUniversity of MagdeburgMagdeburgGermany
  3. 3.Department of Psychosomatic Medicine and PsychotherapyUlm UniversityUlmGermany

Personalised recommendations