Spatiotemporal Facial Super-Pixels for Pain Detection

  • Dennis H. Lundtoft
  • Kamal Nasrollahi
  • Thomas B. Moeslund
  • Sergio Escalera
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9756)


Pain detection using facial images is of critical importance in many Health applications. Since pain is a spatiotemporal process, recent works on this topic employ facial spatiotemporal features to detect pain. These systems extract such features from the entire area of the face. In this paper, we show that by employing super-pixels we can divide the face into three regions, in a way that only one of these regions (about one third of the face) contributes to the pain estimation and the other two regions can be discarded. The experimental results on the UNBC-McMaster database show that the proposed system using this single region outperforms state-of-the-art systems in detecting no-pain scenarios, while it reaches comparable results in detecting weak and severe pain scenarios.


Facial images Super-pixels Spatiotemporal filters Pain detection 


  1. 1.
    Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., Susstrunk, S.: Slic superpixels compared to state-of-the-art superpixel methods. IEEE Trans. Pattern Anal. Mach. Intell. 34(11), 2274–2282 (2012)CrossRefGoogle Scholar
  2. 2.
    Ashraf, A.B., Lucey, S., Cohn, J.F., Chen, T., Ambadar, Z., Prkachin, K.M., Solomon, P.E.: The painful face âĂŞ pain expression recognition using active appearance models. Image Vis. Comput. 27(12), 1788–1796 (2009). visual and multimodal analysis of human spontaneous behaviourCrossRefGoogle Scholar
  3. 3.
    Brahnam, S., Chuang, C.F., Shih, F.Y., Slack, M.R.: Machine recognition and representation of neonatal facial displays of acute pain. Artif. Intell. Med. 36(3), 211–222 (2006)CrossRefGoogle Scholar
  4. 4.
    Chen, Z., Ansari, R., Wilkie, D.J.: Automated detection of pain from facial expressions: a rule-based approach using aam. In: SPIE Medical Imaging, p. 83143O. International Society for Optics and Photonics (2012)Google Scholar
  5. 5.
    Derpanis, K., Gryn, J.: Three-dimensional nth derivative of gaussian separable steerable filters. In: IEEE International Conference on Image Processing, 2005, ICIP 2005, vol. 3, pp. III-553–III-556, September 2005Google Scholar
  6. 6.
    Gholami, B., Haddad, W.M., Tannenbaum, A.R.: Agitation and pain assessment using digital imaging. In: Annual International Conference of the IEEE Engineering in Medicine and Biology Society 2009, EMBC 2009, pp. 2176–2179. IEEE (2009)Google Scholar
  7. 7.
    Hammal, Z., Cohn, J.F.: Automatic detection of pain intensity. In: Proceedings of the 14th ACM International Conference on Multimodal Interaction. pp. 47–52. ACM (2012)Google Scholar
  8. 8.
    Irani, R., Nasrollahi, K., Moeslund, T.B.: Pain recognition using spatiotemporal oriented energy of facial muscles. In: 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshop, pp. 679–692 (2015)Google Scholar
  9. 9.
    Irani, R., Nasrollahi, K., Simon, M.O., Corneanu, C.A., Escalera, S., Bahnsen, C., Lundtoft, D.H., Moeslund, T.B., Pedersen, T.L., Klitgaard, M.L., et al.: Spatiotemporal analysis of RGB-DT facial images for multimodal pain level recognition (2015)Google Scholar
  10. 10.
    Kaltwang, S., Rudovic, O., Pantic, M.: Continuous pain intensity estimation from facial expressions. In: Bebis, G., Boyle, R., Parvin, B., Koracin, D., Fowlkes, C., Wang, S., Choi, M.-H., Mantler, S., Schulze, J., Acevedo, D., Mueller, K., Papka, M. (eds.) ISVC 2012, Part II. LNCS, vol. 7432, pp. 368–377. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  11. 11.
    Littlewort, G.C., Bartlett, M.S., Lee, K.: Automatic coding of facial expressions displayed during posed and genuine pain. Image Vis. Comput. 27(12), 1797–1803 (2009)CrossRefGoogle Scholar
  12. 12.
    Lucey, P., Cohn, J.F., Prkachin, K.M., Solomon, P.E., Chew, S., Matthews, I.: The UNBC-McMaster Shoulder Pain Expression Archive Database (2011). link to UNBC-MacMaster Shoulder Pain DatabaseGoogle Scholar
  13. 13.
    Lucey, P., Cohn, J.F., Prkachin, K.M., Solomon, P.E., Chew, S., Matthews, I.: Painful monitoring: automatic pain monitoring using the unbc-mcmaster shoulder pain expression archive database. Image Vis. Comput. 30(3), 197–205 (2012)CrossRefGoogle Scholar
  14. 14.
    Monwar, M., Rezaei, S.: Appearance-based pain recognition from video sequences. In: International Joint Conference on Neural Networks, 2006, IJCNN 2006, pp. 2429–2434 (2006)Google Scholar
  15. 15.
    Sikka, K., Dhall, A., Bartlett, M.: Weakly supervised pain localization using multiple instance learning. In: 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pp. 1–8. IEEE (2013)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Dennis H. Lundtoft
    • 1
  • Kamal Nasrollahi
    • 1
  • Thomas B. Moeslund
    • 1
  • Sergio Escalera
    • 1
  1. 1.Aalborg UniversityAalborgDenmark

Personalised recommendations