Skip to main content
SpringerLink
Log in

An affect-based classification of emotions associated with images of food

  • Original Paper
  • Published:
Journal of Food Measurement and Characterization Aims and scope Submit manuscript

Abstract

Food and emotions are correlated. Recent research on the relationship between foods and emotions mainly focused on identifying emotions when viewing food images. The studies try to find image attributes that evoke food-related emotions. We concentrate on affective image classification and investigate the performance of different features in a food-related emotion classification framework. First, we extract features of different levels for each food image. Very basic low-level features and art features derived from principle-of-art features are extracted as mid-level features. Then, we develop models for valence-arousal affect dimensions trained using different machine learning techniques. Extensive experiments are conducted on a combined food image dataset. The results demonstrate the effectiveness of the proposed food-related emotion classification method. The results demonstrate the effectiveness of the proposed food-related emotion classification model by comparing different classifiers for the two affect dimensions (valence and arousal), resulting in an accuracy of 67% and 88% respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Abbreviations

AUC:

Area under the curve

CNN:

Convolutional neural networks

IVT:

Intelligent vehicular technologies

MLP:

Multilayer perceptron

PR:

Precision-recall

QDA:

Quadratic discriminant analysis

RTV:

Relative total variation

SAGR:

Sign agreement metric

SIFT:

Scale-invariant feature transform

SVM:

Linear support vector machine

TPR:

True positive rate

VAS:

Visual analogue scale

WTA:

Winner-takes-all

References

  1. P. Robinson, T. Baltrusaitis, I. Davies, T. Pfister, L. Rick, K. Hull, in Proceedings of the 2011 International Conference on Pervasive Computing (2011)

  2. L.N. van der Laan, D.T. De Ridder, M.A. Viergever, P.A. Smeets, Neuroimage 55(1), 296 (2011)

    PubMed  Google Scholar 

  3. Q.L. Leong, S. Karim, Handbook of Research on Global Hospitality and Tourism Management (IGI Global, Pennsylvania, 2015), pp. 406–439

    Google Scholar 

  4. L.Q. Ling, M.S.A. Karim, M. Othman, N.M. Adzahan, S. Ramachandran, World Appl. Sci. J. 10, 164 (2010)

    Google Scholar 

  5. S. Seo, N. Yun, O.Y. Kim, Curr. Issues Tourism 20(2), 135 (2017)

    Google Scholar 

  6. D.A. Zellner, C.R. Loss, J. Zearfoss, S. Remolina, Appetite 77, 31 (2014)

    PubMed  Google Scholar 

  7. L. Barthomeuf, S. Rousset, S. Droit-Volet, Appetite 52(1), 27 (2009)

    CAS  PubMed  Google Scholar 

  8. E.P. Köster, J. Mojet, Food Res. Int. 76(2), 180 (2015)

    Google Scholar 

  9. K. Gallo, Understanding children’s food-related emotions using words and emojis in the united states and ghana. Ph.D. thesis, Kansas State University (2016)

  10. M. Borgogno, A.V. Cardello, S. Favotto, E. Piasentier, Meat Sci. 127, 1 (2017)

    PubMed  Google Scholar 

  11. C.Y. Huang, T.H.K. Huang, L.W. Ku, in Proceedings of the 2017 AAAI Spring Symposium Series (2017)

  12. I. Lopatovska, in Proceedings of the 2011 iConference (ACM, 2011), pp. 225–229

  13. B.W. Schuller, IEEE Ann. Hist. Comput. 1, 1 (2017)

    Google Scholar 

  14. Y. Yang, J. Jia, S. Zhang, B. Wu, Q. Chen, J. Li, C. Xing, J. Tang, in Proceedings of the 28th AAAI Conference on Artificial Intelligence (AAAI Press, 2014), AAAI’14, pp. 306–312

  15. Y. Yang, J. Jia, B. Wu, J. Tang, in Proceedings of the 13th AAAI Conference on Artificial Intelligence (2016)

  16. S.M. Srinivasan, R.S. Sangwan, C.J. Neill, T. Zu, I.E.E.E. Technol, Soc. Mag. 38(1), 58 (2019)

    Google Scholar 

  17. K. Kang, C. Yoon, E.Y. Kim, in Proceedings of the 2016 International Conference on Big Data and Smart Computing (IEEE Computer Society, 2016), pp. 231–238

  18. J.C.S. Hung, K.H. Chiang, Y.H. Huang, K.C. Lin, Multimed. Tools Appl. 76(18), 18361 (2017)

    Google Scholar 

  19. E. Geslin, L. Jégou, D. Beaudoin, Int. J. Comput. Games Technol. 2016, 1 (2016)

    Google Scholar 

  20. J.A. Bopp, E.D. Mekler, K. Opwis, in Proceedings of the 2016 Conference on Human Factors in Computing Systems (ACM, 2016), pp. 2996–3006

  21. Y.G. Jiang, B. Xu, X. Xue, in Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence (AAAI Press, 2014), AAAI’14, pp. 73–79

  22. Y.L. Chen, C.L. Chang, C.S. Yeh, Decis. Support Syst. 101, 40 (2017)

    Google Scholar 

  23. L. Devillers, L. Vidrascu, L. Lamel, Neural Netw. 18(4), 407 (2005)

    PubMed  Google Scholar 

  24. M.S. Bilkhu, S. Gupta, V.K. Srivastava, in Computational Intelligence: Theories, Applications and Future Directions-Volume II (Springer, 2019), pp. 585–594

  25. K.H.Y. Lin, C. Yang, H.H. Chen, in Proceedings of the 2008 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (IEEE Computer Society, 2008), WI-IAT ’08, pp. 220–226

  26. R. Meo, E. Sulis, A.C.M. Trans, Internet Technol. 17(1), 1 (2017)

    Google Scholar 

  27. K. Ravi, B. Manisha, G. Sravan, B. Venkatesh, K.S. Priyanka, Indian J. Sci. Technol. 9, 30 (2016)

    Google Scholar 

  28. Q. Xiao, C. Hu, G. Ding, in International Conference on Swarm Intelligence (Springer, 2017), pp. 608–614

  29. J. Machajdik, A. Hanbury, in Proceedings of the 18th ACM international conference on Multimedia (ACM, 2010), pp. 83–92

  30. S. Zhao, Y. Gao, X. Jiang, H. Yao, T.S. Chua, X. Sun, in Proceedings of the 22nd ACM international conference on Multimedia (ACM, 2014), pp. 47–56

  31. J. Frommel, C. Schrader, M. Weber, in Proceedings of the 2018 Annual Symposium on Computer-Human Interaction Play (ACM, 2018), pp. 173–185

  32. Q. Wu, C. Zhou, C. Wang, in International Conference on Affective Computing and Intelligent Interaction (Springer, 2005), pp. 239–247

  33. M. Sarma, K. Bhattacharyya, J. Eng. Technol. 4, 134 (2016)

    Google Scholar 

  34. W. Wei, Q. Jia, G. Chen, in Proceedings of the 11th IEEE Conference on Industrial Electronics and Applications (IEEE, 2016), pp. 161–165

  35. W.H. Abdulsalam, R.S. Alhamdani, M.N. Abdullah, Int. J. Mach. Learn. Comput. 9, 1 (2019)

    Google Scholar 

  36. P.O. Staff, PLoS ONE 14(3), e0213756 (2019)

    Google Scholar 

  37. X. Liu, N. Li, Y. Xia, J. Vis. Commun. Image Represent. 58, 576 (2019)

    Google Scholar 

  38. M.S. Hossain, G. Muhammad, Inf. Fusion 49, 69 (2019)

    Google Scholar 

  39. G. Ares, A. Giménez, L. Vidal, Y. Zhou, A. Krystallis, G. Tsalis, R. Symoneaux, L.M. Cunha, A.P. de Moura, A. Claret et al., Food Qual. Prefer. 52, 62 (2016)

    Google Scholar 

  40. R. Barrena, T. García, M. Sánchez, Agrekon 56, 1–18 (2017)

    Google Scholar 

  41. A.M. Brouwer, M. Hogervorst, M. Grootjen, J. van Erp, E. Zandstra, Food Qual. Preference 62(Supplement C), 307 (2017)

    Google Scholar 

  42. E. Bartkiene, V. Steibliene, V. Adomaitiene, G. Juodeikiene, D. Cernauskas, V. Lele, D. Klupsaite, D. Zadeike, L. Jarutiene, R.P. Guiné, BioMed Res. Int. 2019, 8739375 (2019)

    Google Scholar 

  43. E. Robinson, J. Henderson, G.S. Keenan, I. Kersbergen, Food Qual. Preference 75, 113 (2019)

    Google Scholar 

  44. L. Jampour, S. Jafarirad, B. Cheraghian, F. Behrouzian, Nutr. Today 54(2), 88 (2019)

    Google Scholar 

  45. L. Mattar, N. Farran, J.A. Kharma, N. Zeeni, Eat. Behav. 33, 7 (2019)

    CAS  PubMed  Google Scholar 

  46. C. Padulo, L. Carlucci, V. Manippa, D. Marzoli, A. Saggino, L. Tommasi, S. Puglisi-Allegra, A. Brancucci, Food Qual. Preference 57, 104 (2017)

    Google Scholar 

  47. G.J. Privitera, M. Diaz, M.C. Haas, Glob. J. Health Sci. 6(3), 1 (2014)

    PubMed  PubMed Central  Google Scholar 

  48. S. Gutjar, C. de Graaf, V. Kooijman, R.A. de Wijk, A. Nys, G.J. Ter Horst, G. Jager, Food Res. Int. 76, 216 (2015)

    Google Scholar 

  49. P. van Esch, J. Heller, G. Northey, J. Retail. Consum. Serv. 50, 94 (2019)

    Google Scholar 

  50. J.Y. Kim, J. Prescott, K.O. Kim, Food Qual. Preference 59, 1 (2017)

    Google Scholar 

  51. D.R. Evans, Psychon. Sci. 23(4), 303 (1971)

    Google Scholar 

  52. A. Lang, P. Bolls, K. Kawahara, in. Proceedings of the Midwest Artificial and Cognitive Science Conferece Intelligence (1996), p. 1996

  53. I. Van Den Bosch, V. Salimpoor, R.J. Zatorre, Front. Hum. Neurosci. 7, 534 (2013)

    PubMed  PubMed Central  Google Scholar 

  54. L. Miccoli, R. Delgado, P. Guerra, F. Versace, S. Rodríguez-Ruiz, M.C. Fernández-Santaella, PLoS ONE 11(8), 1 (2016)

    Google Scholar 

  55. B.L. Carter, C.Y. Lam, J.D. Robinson, M.M. Paris, A.J. Waters, D.W. Wetter, P.M. Cinciripini, Nicotine Tob. Res. 11(7), 823 (2009)

    PubMed  PubMed Central  Google Scholar 

  56. A. Schacht, A. Łuczak, T. Pinkpank, T. Vilgis, W. Sommer, Int. J. Gastron. Food Sci. 5, 33 (2016)

    Google Scholar 

  57. M. Frayn, C.R. Sears, K.M. von Ranson, Appetite 100, 55 (2016)

    PubMed  Google Scholar 

  58. X. Hou, L. Zhang, in Proceedings of the 2007 IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2007), pp. 1–8

  59. J.L. Pech-Pacheco, G. Cristóbal, J. Chamorro-Martinez, J. Fernández-Valdivia, in Proceedings of the 15th International Conference on Pattern Recognition, vol. 3 (IEEE, 2000), vol. 3, pp. 314–317

  60. J. Blechert, A. Lender, S. Polk, N.A. Busch, K. Ohla, Front. Psychol. 10, 307 (2019)

    PubMed  PubMed Central  Google Scholar 

  61. F. Foroni, G. Pergola, G. Argiris, R.I. Rumiati, Front. Hum. Neurosci. 7, 51 (2013)

    PubMed  PubMed Central  Google Scholar 

  62. A. Toet, D. Kaneko, I. de Kruijf, S. Ushiama, M.G. van Schaik, A.M. Brouwer, V. Kallen, J.B. Van Erp, Front. Psychol. 10, 58 (2019)

    PubMed  PubMed Central  Google Scholar 

  63. M.H. McManis, M.M. Bradley, W.K. Berg, B.N. Cuthbert, P.J. Lang, Psychophysiology 38(2), 222 (2001)

    CAS  PubMed  Google Scholar 

  64. M.M. Bradley, P.J. Lang, Cognit. Neurosci. Emot. 25, 49 (2000)

    Google Scholar 

  65. A. Yazdani, E. Skodras, N. Fakotakis, T. Ebrahimi, EURASIP J. Image Video Process. 2013(1), 26 (2013)

    Google Scholar 

  66. M.A. Nicolaou, H. Gunes, M. Pantic, IEEE Trans. Affect. Comput. 2(2), 92 (2011)

    Google Scholar 

  67. J.H. Kim, H. Youn, Y. Rao, Int. J. Hosp. Manag. 61, 129 (2017)

    Google Scholar 

  68. L. Miccoli, R. Delgado, S. Rodríguez-Ruiz, P. Guerra, E. García-Mármol, M.C. Fernández-Santaella, PLoS ONE 9(12), 1 (2014)

    Google Scholar 

  69. L. Charbonnier, F. van Meer, L.N. van der Laan, M.A. Viergever, P.A. Smeets, Appetite 96, 166 (2016)

    PubMed  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Electrical Engineering and Computer Science (SEECS), National University of Sciences and Technology (NUST), Islamabad, Pakistan

    Yusra Tahir & Anis Ur Rahman

  2. Department of Information Systems, Faculty of Computer Science and Information Technology, University of Malaya, Kuala Lumpur, Malaysia

    Sri Devi Ravana

Authors
  1. Yusra Tahir
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anis Ur Rahman
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Sri Devi Ravana
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Anis Ur Rahman.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Tahir, Y., Rahman, A.U. & Ravana, S.D. An affect-based classification of emotions associated with images of food. Food Measure 15, 519–530 (2021). https://doi.org/10.1007/s11694-020-00650-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11694-020-00650-7

Keywords

Search

Navigation