The Color of Smiling: Computational Synaesthesia of Facial Expressions

  • Vittorio CuculoEmail author
  • Raffaella Lanzarotti
  • Giuseppe Boccignone
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9279)


This note gives a preliminary account of the transcoding or rechanneling problem between different stimuli as it is of interest for the natural interaction or affective computing fields. By the consideration of a simple example, namely the color response of an affective lamp to a sensed facial expression, we frame the problem within an information-theoretic perspective. A full justification in terms of the Information Bottleneck principle promotes a latent affective space, hitherto surmised as an appealing and intuitive solution, as a suitable mediator between the different stimuli.


Affective computing Facial expressions Information-bottleneck Graphical models 


  1. 1.
    Adamo, A., Grossi, G., Lanzarotti, R.: Local features and sparse representation for face recognition with partial occlusions. In: 20th IEEE Int. Conf. on Image Processing (ICIP), pp. 3008–3012, September 2013Google Scholar
  2. 2.
    Bialek, W., van Steveninck, R.R.R., Tishby, N.: Efficient representation as a design principle for neural coding and computation. In: IEEE International Symposium on Information Theory, pp. 659–663. IEEE (2006)Google Scholar
  3. 3.
    Birren, F.: Color Psychology and Color Therapy. Kessinger Publishing (2006)Google Scholar
  4. 4.
    Broekens, J., Brinkman, W.: Affectbutton: A method for reliable and valid affective self-report. Int. J. Hum.-Comput. Stud. 71(6), 641–667 (2013)CrossRefGoogle Scholar
  5. 5.
    Chechik, G., Globerson, A., Tishby, N., Weiss, Y.: Information bottleneck for gaussian variables. Journal of Machine Learning Research 6, 165–188 (2005)MathSciNetzbMATHGoogle Scholar
  6. 6.
    Collier, G.L.: Affective synesthesia: Extracting emotion space from simple perceptual stimuli. Motivation and emotion 20(1), 1–32 (1996)CrossRefGoogle Scholar
  7. 7.
    Cover, T., Thomas, J.: Elements of Information Theory. Wiley and Sons, New York (1991)CrossRefzbMATHGoogle Scholar
  8. 8.
    Cuculo, V., Lanzarotti, R., Boccignone, G.: Using sparse coding for landmark localization in facial expressions. In: 5th European Workshop on Visual Information Processing (EUVIP), pp. 1–6, December 2014Google Scholar
  9. 9.
    Ekman, P.: Facial expression and emotion. American Psychologist 48(4), 384 (1993)CrossRefGoogle Scholar
  10. 10.
    Elidan, G., Friedman, N.: Learning hidden variable networks: The information bottleneck approach. J. of Machine Learning Research 6, 81–127 (2005)MathSciNetzbMATHGoogle Scholar
  11. 11.
    Friedman, N., Mosenzon, O., Slonim, N., Tishby, N.: Multivariate information bottleneck. In: Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence, pp. 152–161 (2001)Google Scholar
  12. 12.
    Gao, X.P., Xin, J.: Investigation of human’s emotional responses on colors. Color Research & Application 31(5), 411–417 (2006)CrossRefGoogle Scholar
  13. 13.
    Hoffmann, H., Scheck, A., Schuster, T., Walter, S., Limbrecht, K., Traue, H.C., Kessler, H.: Mapping discrete emotions into the dimensional space: An empirical approach. In: IEEE International Conference on Systems, Man, and Cybernetics (SMC), pp. 3316–3320. IEEE (2012)Google Scholar
  14. 14.
    Jeni, L., Girard, J., Cohn, J., De la Torre, F.: Continuous AU intensity estimation using localized, sparse facial feature space. In: 10th IEEE Int. Conf. and Workshops Automat. Face Gesture Recogn., pp. 1–7, April 2013Google Scholar
  15. 15.
    Kim, M., Lee, H., Park, J., Jo, S., Chung, M.: Determining color and blinking to support facial expression of a robot for conveying emotional intensity. In: Int’l Symposium on Robot and Human Interactive Communication, RO-MAN, pp. 219–24 (2008)Google Scholar
  16. 16.
    Mahnke, F.H.: COLOR, Environment, & Human Response. John Wiley & Sons (1996)Google Scholar
  17. 17.
    Murphy, K.P.: Machine learning: a probabilistic perspective. MIT Press, Cambridge (2012)Google Scholar
  18. 18.
    Osgood, C.E., Suci, G.J., Tannenbaum, P.H.: The measurement of meaning. University of Illinois Press (1964)Google Scholar
  19. 19.
    Pessoa, L.: On the relationship between emotion and cognition. Nature Reviews Neuroscience 9(2), 148–158 (2008)CrossRefGoogle Scholar
  20. 20.
    Plutchik, R.: Emotion: Theory. Research and Experience. Acad. Pr. (1980)Google Scholar
  21. 21.
    Russell, J.A.: Core affect and the psychological construction of emotion. Psychological Review 110(1), 145 (2003)CrossRefGoogle Scholar
  22. 22.
    Russell, J.A., Mehrabian, A.: Evidence for a three-factor theory of emotions. Journal of Research in Personality 11(3), 273–294 (1977)CrossRefGoogle Scholar
  23. 23.
    Scheirer, J., Picard, R.: Affective objects. MIT Media lab Technical Rep. 524 (2000)Google Scholar
  24. 24.
    Slonim, N., Weiss, Y.: Maximum likelihood and the information bottleneck. In: Advances in Neural Information Processing Systems, pp. 335–342 (2002)Google Scholar
  25. 25.
    Spence, C.: Crossmodal correspondences: A tutorial review. Attention, Perception, & Psychophysics 73(4), 971–995 (2011)MathSciNetCrossRefGoogle Scholar
  26. 26.
    Suk, H.J., Irtel, H.: Emotional response to color across media. Color Research & Application 35(1), 64–77 (2010)CrossRefGoogle Scholar
  27. 27.
    Tishby, N., Pereira, F.C., Bialek, W.: The information bottleneck method. In: The 37th Allerton Conference on Communication, Control, and Computing (1999)Google Scholar
  28. 28.
    Valdez, P., Mehrabian, A.: Effects of color on emotions. Journal of Experimental Psychology: General 123(4), 394 (1994)CrossRefGoogle Scholar
  29. 29.
    Vinciarelli, A., Pantic, M., Heylen, D., Pelachaud, C., Poggi, I., D’Errico, F., Schroeder, M.: Bridging the gap between social animal and unsocial machine: A survey of social signal processing 3(1), 69–87 (2012)Google Scholar
  30. 30.
    Vitale, J., Williams, M.A., Johnston, B., Boccignone, G.: Affective facial expression processing via simulation: A probabilistic model. Biologically Inspired Cognitive Architectures Journal 10, 30–41 (2014)CrossRefGoogle Scholar
  31. 31.
    Ward, J.: Emotionally mediated synaesthesia. Cognitive Neuropsychology 21(7), 761–772 (2004)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Vittorio Cuculo
    • 1
    Email author
  • Raffaella Lanzarotti
    • 2
  • Giuseppe Boccignone
    • 2
  1. 1.Dipartimento di MatematicaUniversità di MilanoMilanoItaly
  2. 2.Dipartimento di InformaticaUniversità di MilanoMilanoItaly

Personalised recommendations