Spatio-temporal Pain Recognition in CNN-Based Super-Resolved Facial Images

  • Marco BellantonioEmail author
  • Mohammad A. Haque
  • Pau Rodriguez
  • Kamal Nasrollahi
  • Taisi Telve
  • Sergio Escalera
  • Jordi Gonzalez
  • Thomas B. Moeslund
  • Pejman Rasti
  • Gholamreza Anbarjafari
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10165)


Automatic pain detection is a long expected solution to a prevalent medical problem of pain management. This is more relevant when the subject of pain is young children or patients with limited ability to communicate about their pain experience. Computer vision-based analysis of facial pain expression provides a way of efficient pain detection. When deep machine learning methods came into the scene, automatic pain detection exhibited even better performance. In this paper, we figured out three important factors to exploit in automatic pain detection: spatial information available regarding to pain in each of the facial video frames, temporal axis information regarding to pain expression pattern in a subject video sequence, and variation of face resolution. We employed a combination of convolutional neural network and recurrent neural network to setup a deep hybrid pain detection framework that is able to exploit both spatial and temporal pain information from facial video. In order to analyze the effect of different facial resolutions, we introduce a super-resolution algorithm to generate facial video frames with different resolution setups. We investigated the performance on the publicly available UNBC-McMaster Shoulder Pain database. As a contribution, the paper provides novel and important information regarding to the performance of a hybrid deep learning framework for pain detection in facial images of different resolution.


Super-Resolution Convolutional Neural Network (CNN) Recurrent Neural Network (RNN) Pain detection 


  1. 1.
    Capel, D., Zisserman, A.: Super-resolution enhancement of text image sequences. In: 2000 Proceedings of the 15th International Conference on Pattern Recognition, vol. 1, pp. 600–605. IEEE (2000)Google Scholar
  2. 2.
    Craig, K.D., Prkachin, K.M., Grunau, R.E.: The facial expression of pain. In: Handbook of Pain Assessment. Guilford Press (2011)Google Scholar
  3. 3.
    Craig, K.D., Hyde, S.A., Patrick, C.J.: Genuine, suppressed and faked facial behavior during exacerbation of chronic low back pain. Pain 46(2), 161–171 (1991)CrossRefGoogle Scholar
  4. 4.
    Cristani, M., Cheng, D.S., Murino, V., Pannullo, D.: Distilling information with super-resolution for video surveillance. In: Proceedings of the ACM 2nd International Workshop on Video Surveillance & Sensor Networks, pp. 2–11. ACM (2004)Google Scholar
  5. 5.
    Debono, D.J., Hoeksema, L.J., Hobbs, R.D.: Caring for patients with chronic pain: pearls and pitfalls. J. Am. Osteopath. Assoc. 113(8), 620–627 (2013). doi: 10.7556/jaoa.2013.023 CrossRefGoogle Scholar
  6. 6.
    Ekman, P., Friesen, W.: Facial Action Coding System: A Technique for the Measurement of Facial Movement. Consulting Psychologists Press, Palo Alto (1978)Google Scholar
  7. 7.
    Freeman, W.T., Jones, T.R., Pasztor, E.C.: Example-based super-resolution. IEEE Comput. Graphics Appl. 22(2), 56–65 (2002)CrossRefGoogle Scholar
  8. 8.
    Gers, F.A., Schmidhuber, J.A., Cummins, F.A.: Learning to forget: continual prediction with LSTM. Neural Comput. 12(10), 2451–2471 (2000)CrossRefGoogle Scholar
  9. 9.
    Gunturk, B.K., Batur, A.U., Altunbasak, Y., Hayes, M.H., Mersereau, R.M.: Eigenface-domain super-resolution for face recognition. IEEE Trans. Image Process. 12(5), 597–606 (2003)CrossRefGoogle Scholar
  10. 10.
    Hadjistavropoulos, T., LaChapelle, D.L., MacLeod, F.K., Snider, B., Craig, K.D.: Measuring movement-exacerbated pain in cognitively impaired frail elders. 16, 54–63 (2000)Google Scholar
  11. 11.
    Haque, M.A., Nasrollahi, K., Moeslund, T.B.: Real-time acquisition of high quality face sequences from an active pan-tilt-zoom camera. In: 2013 10th IEEE International Conference on Advanced Video and Signal Based Surveillance, pp. 443–448, August 2013Google Scholar
  12. 12.
    Haque, M.A., Nasrollahi, K., Moeslund, T.B.: Constructing facial expression log from video sequences using face quality assessment. In: 2014 International Conference on Computer Vision Theory and Applications (VISAPP), vol. 2, pp. 517–525, January 2014Google Scholar
  13. 13.
    Haque, M.A., Nasrollahi, K., Moeslund, T.B.: Quality-aware estimation of facial landmarks in video sequences. In: 2015 IEEE Winter Conference on Applications of Computer Vision, pp. 678–685, January 2015Google Scholar
  14. 14.
    Haque, M.A., Nasrollahi, K., Moeslund, T.B.: Pain expression as a biometric: why patients’ self-reported pain doesn’t match with the objectively measured pain? In: 2017 IEEE International Conference on Identity, Security and Behavior Analysis (ISBA), February 2017 (submitted)Google Scholar
  15. 15.
    Hennings-Yeomans, P.H., Baker, S., Kumar, B.V.: Recognition of low-resolution faces using multiple still images and multiple cameras. In: 2008 2nd IEEE International Conference on Biometrics: Theory, Applications and Systems, BTAS 2008, pp. 1–6. IEEE (2008)Google Scholar
  16. 16.
    Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)CrossRefGoogle Scholar
  17. 17.
    Huang, T.S., Tsay, R.Y.: Multiple frame image restoration and registration. In: Advances in Computer Vision and Image Processing, pp. 317–339 (1984)Google Scholar
  18. 18.
    Huang, Z., Wang, R., Shan, S., Chen, X.: Face recognition on large-scale video in the wild with hybrid Euclidean-and-Riemannian metric learning. Pattern Recogn. 48(10), 3113–3124 (2015)CrossRefGoogle Scholar
  19. 19.
    Irani, R., Nasrollahi, K., Moeslund, T.B.: Pain recognition using spatiotemporal oriented energy of facial muscles. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 80–87, June 2015Google Scholar
  20. 20.
    Irani, R., Nasrollahi, K., Simon, M.O., Corneanu, C.A., Escalera, S., Bahnsen, C., Lundtoft, D.H., Moeslund, T.B., Pedersen, T.L., Klitgaard, M.L., Petrini, L.: Spatiotemporal analysis of RGB-D-T facial images for multimodal pain level recognition. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2015Google Scholar
  21. 21.
    Kahou, S.E., Bouthillier, X., Lamblin, P., Gulcehre, C., Michalski, V., Konda, K., Jean, S., Froumenty, P., Dauphin, Y., Boulanger-Lewandowski, N., Chandias Ferrari, R., Mirza, M., Warde-Farley, D., Courville, A., Vincent, P., Memisevic, R., Pal, C., Bengio, Y.: Emonets: multimodal deep learning approaches for emotion recognition in video. J. Multimodal User Interfaces 10(2), 99–111 (2016)CrossRefGoogle Scholar
  22. 22.
    Kennedy, J.A., Israel, O., Frenkel, A., Bar-Shalom, R., Azhari, H.: Super-resolution in pet imaging. IEEE Trans. Med. Imaging 25(2), 137–147 (2006)CrossRefGoogle Scholar
  23. 23.
    Khorrami, P., Paine, T.L., Brady, K., Dagli, C., Huang, T.S.: How deep neural networks can improve emotion recognition on video data. In: 2016 IEEE International Conference on Image Processing (ICIP), pp. 619–623, September 2016Google Scholar
  24. 24.
    Kim, B.K., Roh, J., Dong, S.Y., Lee, S.Y.: Hierarchical committee of deep convolutional neural networks for robust facial expression recognition. J. Multimodal User Interfaces 10(2), 173–189 (2016)CrossRefGoogle Scholar
  25. 25.
    Kim, K.I., Kim, D., Kim, J.H.: Example-based learning for image super-resolution (2004)Google Scholar
  26. 26.
    Kim, K.I., Kwon, Y.: Example-based learning for single-image super-resolution. In: Rigoll, G. (ed.) DAGM 2008. LNCS, vol. 5096, pp. 456–465. Springer, Heidelberg (2008). doi: 10.1007/978-3-540-69321-5_46 CrossRefGoogle Scholar
  27. 27.
    Kingma, D., Ba, J.: Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  28. 28.
    Kunz, M., Gruber, A., Lautenbacher, S.: Sex differences in facial encoding of pain. J. Pain 7(12), 915–928 (2006)CrossRefGoogle Scholar
  29. 29.
    Kunz, M., Mylius, V., Schepelmann, K., Lautenbacher, S.: On the relationship between self-report and facial expression of pain. J. Pain 5(7), 368–376 (2004)CrossRefGoogle Scholar
  30. 30.
    Kunz, M., Prkachin, K., Lautenbacher, S.: Smiling in pain: explorations of its social motives. Pain Res. Treat. 2013, e128093 (2013)Google Scholar
  31. 31.
    Kunz, M., Scharmann, S., Hemmeter, U., Schepelmann, K., Lautenbacher, S.: The facial expression of pain in patients with dementia. Pain 133(1–3), 221–228 (2007)CrossRefGoogle Scholar
  32. 32.
    Lautenbacher, S., Niewelt, B.G., Kunz, M.: Decoding pain from the facial display of patients with dementia: a comparison of professional and nonprofessional observers. Pain Med. 14(4), 469–477 (2013). CrossRefGoogle Scholar
  33. 33.
    Léonard, N., Waghmare, S., Wang, Y.: Rnn: Recurrent library for torch. arXiv preprint arXiv:1511.07889 (2015)
  34. 34.
    Li, F., Jia, X., Fraser, D.: Universal HMT based super resolution for remote sensing images. In: 2008 15th IEEE International Conference on Image Processing, ICIP 2008, pp. 333–336. IEEE (2008)Google Scholar
  35. 35.
    Li, H., Hua, G.: Hierarchical-PEP model for real-world face recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, pp. 4055–4064 (2015)Google Scholar
  36. 36.
    Lin, F.C., Fookes, C.B., Chandran, V., Sridharan, S.: Investigation into optical flow super-resolution for surveillance applications (2005)Google Scholar
  37. 37.
    Lucey, P., Cohn, J.F., Matthews, I., Lucey, S., Sridharan, S., Howlett, J., Prkachin, K.M.: Automatically detecting pain in video through facial action units. IEEE Trans. Syst. Man Cybern. Part B (Cybern.) 41(3), 664–674 (2011)CrossRefGoogle Scholar
  38. 38.
    Lucey, P., Cohn, J.F., Prkachin, K.M., Solomon, P.E., Matthews, I.: Painful data: the UNBC-McMaster shoulder pain expression archive database. In: 2011 IEEE International Conference on Automatic Face Gesture Recognition and Workshops (FG 2011), pp. 57–64, March 2011Google Scholar
  39. 39.
    Maintz, J.A., Viergever, M.A.: A survey of medical image registration. Med. Image Anal. 2(1), 1–36 (1998)CrossRefGoogle Scholar
  40. 40.
    Malczewski, K., Stasinski, R.: Toeplitz-based iterative image fusion scheme for MRI. In: 2008 15th IEEE International Conference on Image Processing, ICIP 2008, pp. 341–344. IEEE (2008)Google Scholar
  41. 41.
    Milanfar, P.: Super-Resolution Imaging. CRC Press, Boca Raton (2010)Google Scholar
  42. 42.
    Nasrollahi, K., Moeslund, T.B.: Super-resolution: a comprehensive survey. Mach. Vis. Appl. 25(6), 1423–1468 (2014)CrossRefGoogle Scholar
  43. 43.
    Parkhi, O.M., Vedaldi, A., Zisserman, A.: Deep face recognition. In: British Machine Vision Conference, vol. 1, p. 6 (2015)Google Scholar
  44. 44.
    Peled, S., Yeshurun, Y.: Superresolution in MRI: application to human white matter fiber tract visualization by diffusion tensor imaging. Magn. Reson. Med. 45(1), 29–35 (2001)CrossRefGoogle Scholar
  45. 45.
    Prkachin, K.M.: The consistency of facial expressions of pain: a comparison across modalities. 51, 297–306 (1992)Google Scholar
  46. 46.
    Prkachin, K.M., Berzins, S., Mercer, S.R.: Encoding and decoding of pain expressions: a judgement study. Pain 58(2), 253–259 (1994)CrossRefGoogle Scholar
  47. 47.
    Prkachin, K.M., Solomon, P.E.: The structure, reliability and validity of pain expression: Evidence from patients with shoulder pain. 139, 267–274 (2008)Google Scholar
  48. 48.
    Prkachin, K., Schultz, I., Berkowitz, J., Hughes, E., Hunt, D.: Assessing pain behaviour of low-back pain patients in real time: concurrent validity and examiner sensitivity. Behav. Res. Ther. 40(5), 595–607 (2002)CrossRefGoogle Scholar
  49. 49.
    Ranganathan, H., Chakraborty, S., Panchanathan, S.: Multimodal Emotion Recognition Using Deep Learning Architectures. Institute of Electrical and Electronics Engineers Inc., United States (2016)CrossRefGoogle Scholar
  50. 50.
    Sezer, O.G., Altunbasak, Y., Ercil, A.: Face recognition with independent component-based super-resolution. In: Electronic Imaging 2006, pp. 607705–607705. International Society for Optics and Photonics (2006)Google Scholar
  51. 51.
    Sikdar, A., Behera, S.K., Dogra, D.P.: Computer vision guided human pulse rate estimation a review. IEEE Rev. Biomed. Eng. PP(99), 1 (2016)Google Scholar
  52. 52.
    Sikka, K., Ahmed, A.A., Diaz, D., Goodwin, M.S., Craig, K.D., Bartlett, M.S., Huang, J.S.: Automated assessment of children’s postoperative pain using computer vision. Pediatrics 136(1), 124–131 (2015)CrossRefGoogle Scholar
  53. 53.
    Vallerand, A.H., Polomano, R.C.: The relationship of gender to pain. Pain Manage. Nurs. 1(3, Supplement 1), 8–15 (2000)CrossRefGoogle Scholar
  54. 54.
    Yang, J., Huang, T.: Image super-resolution: historical overview and future challenges. In: Super-Resolution Imaging, pp. 20–34 (2010)Google Scholar
  55. 55.
    Yang, J., Ren, P., Chen, D., Wen, F., Li, H., Hua, G.: Neural aggregation network for video face recognition. arXiv preprint arXiv:1603.05474 (2016)
  56. 56.
    Yu, Z., Zhang, C.: Image based static facial expression recognition with multiple deep network learning. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, ICMI 2015, pp. 435–442. ACM, New York (2015)Google Scholar
  57. 57.
    Zhou, J., Hong, X., Su, F., Zhao, G.: Recurrent convolutional neural network regression for continuous pain intensity estimation in video. arXiv preprint arXiv:1605.00894 (2016)

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  • Marco Bellantonio
    • 1
    Email author
  • Mohammad A. Haque
    • 2
  • Pau Rodriguez
    • 1
  • Kamal Nasrollahi
    • 2
  • Taisi Telve
    • 3
  • Sergio Escalera
    • 1
  • Jordi Gonzalez
    • 1
  • Thomas B. Moeslund
    • 2
  • Pejman Rasti
    • 3
  • Gholamreza Anbarjafari
    • 3
  1. 1.Computer Vision Center (UAB)University of BarcelonaBarcelonaSpain
  2. 2.Visual Analysis of People (VAP) LaboratoryAalborg UniversityAalborgDenmark
  3. 3.iCV Research Group, Institute of TechnologyUniversity of TartuTartuEstonia

Personalised recommendations