Abstract
Most of the currently available pain datasets use two types of pain stimuli - people with clinically diagnosed conditions (e.g. surgery) performing tasks that cause them pain (we call this clinical pain) and pain caused by external stimuli such as heat or electricity (we call this experimental pain). In high-risk domains like healthcare, understanding the decisions and limitations of various types of pain recognition models is pivotal for the acceptance of the technology. In this paper, we present a process based on Explainable Artificial Intelligence techniques to investigate the differences in the learned representations of models trained on experimental pain (BioVid heat pain dataset) and clinical pain (UNBC shoulder pain dataset). To this end, we first train two convolutional neural networks - one for each dataset - to automatically discern between pain and no pain. Next, we perform a cross-dataset evaluation, i.e., evaluate the performance of the heat pain model on images from the shoulder pain dataset and vice versa. Then, we use Layer-wise Relevance Propagation to analyze which parts of the images in our test sets were relevant for each pain model. Based on this analysis, we rely on the visual inspection by a human observer to generate hypotheses about learned concepts that distinguish the two models. Finally, we test those hypotheses quantitatively utilizing concept embedding analysis methods. Through this process, we identify (1) a concept which the clinical pain model is more strongly relying on and, (2) a concept which the experimental pain model is paying more attention to. Finally, we discuss how both of these concepts are involved in known pain patterns and can be attributed to behavioral differences found in the datasets.
This work has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 847926 MindBot and from the DFG under project number 392401413, DEEP.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
The final annotations are available upon request to the authors.
References
The biovid heat pain database. http://www.iikt.ovgu.de/BioVid.html. Accessed 18 July 2021
Alber, M., et al.: iNNvestigate neural networks! J. Mach. Learn. Res. 20(93), 1–8 (2019)
André, E., Kunz, M.: Digitale gesichts- bzw. schmerzerkennung und ihr potential für die klinische praxis. In: Digitalisierung und Gesundheit. G.IP - Gesundheitsforschung. Interdisziplinäre Perspektiven, Nomos Verlagsgesellschaft mbH & Co. KG (to appear)
Bau, D., Zhou, B., Khosla, A., Oliva, A., Torralba, A.: Network dissection: quantifying interpretability of deep visual representations. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 3319–3327. IEEE Computer Society (2017)
Cui, Y., Jia, M., Lin, T.Y., Song, Y., Belongie, S.: Class-balanced loss based on effective number of samples. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9268–9277 (2019)
Dai, L., Broekens, J., Truong, K.P.: Real-time pain detection in facial expressions for health robotics. In: 2019 8th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW), pp. 277–283. IEEE (2019)
Dietterich, T.G.: Approximate statistical tests for comparing supervised classification learning algorithms. Neural Comput. 10(7), 1895–1923 (1998)
Khorrami, P., Paine, T., Huang, T.: Do deep neural networks learn facial action units when doing expression recognition? In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 19–27 (2015)
Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., et al.: Interpretability beyond feature attribution: quantitative testing with concept activation vectors (TCAV). In: International Conference on Machine Learning, pp. 2668–2677. PMLR (2018)
Kunz, M., Lautenbacher, S.: The faces of pain: a cluster analysis of individual differences in facial activity patterns of pain. Eur. J. Pain 18(6), 813–823 (2014)
Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 2980–2988 (2017)
Lucey, P., Cohn, J.F., Prkachin, K.M., Solomon, P.E., Matthews, I.: Painful data: the UNBC-McMaster shoulder pain expression archive database. In: Proceedings of the International Conference on Automatic Face & Gesture Recognition and Workshops, pp. 57–64. IEEE (2011)
Mertes, S., Huber, T., Weitz, K., Heimerl, A., André, E.: Ganterfactual - counterfactual explanations for medical non-experts using generative adversarial learning. CoRR abs/2012.11905 (2021)
Mollahosseini, A., Hassani, B., Mahoor, M.H.: AffectNet: a database for facial expression, valence, and arousal computing in the wild. IEEE Trans. Affect. Comput. 10(1), 18–31 (2019)
Montavon, G., Binder, A., Lapuschkin, S., Samek, W., Müller, K.-R.: Layer-wise relevance propagation: an overview. In: Samek, W., Montavon, G., Vedaldi, A., Hansen, L.K., Müller, K.-R. (eds.) Explainable AI: Interpreting, Explaining and Visualizing Deep Learning. LNCS (LNAI), vol. 11700, pp. 193–209. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-28954-6_10
Othman, E., Werner, P., Saxen, F., Al-Hamadi, A., Walter, S.: Cross-database evaluation of pain recognition from facial video. In: 2019 11th International Symposium on Image and Signal Processing and Analysis (ISPA), pp. 181–186. IEEE (2019)
Petyaeva, A., et al.: Feasibility of a staff training and support programme to improve pain assessment and management in people with dementia living in care homes. Int. J. Geriatr. Psychiatry 33(1), 221–231 (2018)
Prajod, P., Schiller, D., Huber, T., André, E.: Do deep neural networks forget facial action units?-exploring the effects of transfer learning in health related facial expression recognition. arXiv preprint arXiv:2104.07389 (2021)
Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should I trust you?”: explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1135–1144 (2016)
Sixt, L., Granz, M., Landgraf, T.: When explanations lie: why many modified BP attributions fail. In: Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13–18 July 2020, Virtual Event, pp. 9046–9057 (2020)
Walter, S., et al.: The biovid heat pain database data for the advancement and systematic validation of an automated pain recognition system. In: 2013 IEEE International Conference on Cybernetics (CYBCO), pp. 128–131. IEEE (2013)
Wang, F., et al.: Regularizing face verification nets for pain intensity regression. In: 2017 IEEE International Conference on Image Processing (ICIP), pp. 1087–1091. IEEE (2017)
Weitz, K., Hassan, T., Schmid, U., Garbas, J.U.: Deep-learned faces of pain and emotions: Elucidating the differences of facial expressions with the help of explainable AI methods. tm-Technisches Messen 86(7–8), 404–412 (2019)
Werner, P., Al-Hamadi, A., Limbrecht-Ecklundt, K., Walter, S., Gruss, S., Traue, H.C.: Automatic pain assessment with facial activity descriptors. IEEE Trans. Affect. Comput. 8(3), 286–299 (2016)
Werner, P., Al-Hamadi, A., Walter, S.: Analysis of facial expressiveness during experimentally induced heat pain. In: 2017 Seventh International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW), pp. 176–180. IEEE (2017)
Zhao, R., Gan, Q., Wang, S., Ji, Q.: Facial expression intensity estimation using ordinal information. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3466–3474 (2016)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2022 Springer Nature Switzerland AG
About this paper
Cite this paper
Prajod, P., Huber, T., André, E. (2022). Using Explainable AI to Identify Differences Between Clinical and Experimental Pain Detection Models Based on Facial Expressions. In: Þór Jónsson, B., et al. MultiMedia Modeling. MMM 2022. Lecture Notes in Computer Science, vol 13141. Springer, Cham. https://doi.org/10.1007/978-3-030-98358-1_25
Download citation
DOI: https://doi.org/10.1007/978-3-030-98358-1_25
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-98357-4
Online ISBN: 978-3-030-98358-1
eBook Packages: Computer ScienceComputer Science (R0)