Understanding Food Images to Recommend Utensils During Meals

Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 10590)

Abstract

Understanding food images can be useful to enable different technologies aimed at improving the quality of life of the society. We focus on the problem of analyzing food images to recognize the utensils to be used to consume the meal depicted in the image. The proposed investigation has both a practical and a theoretical relevance, since (1) it can contribute to the design of intelligent systems able to assist people with mental disabilities and (2) it allows to assess if high level concepts related to food (e.g., how to eat food) can be inferred from visual analysis. We augment the FD1200 dataset with labels related to utensils and perform experiments considering AlexNet features coupled with a multiclass SVM classifier. Results show that, even such a simple classification pipeline can achieve promising results.

Keywords

Image understanding Food analysis Assistive technologies 

References

  1. 1.
    Farinella, G.M., Allegra, D., Moltisanti, M., Stanco, F., Battiato, S.: Retrieval and classification of food images. Comput. Biol. Med. 77, 2339 (2016)CrossRefGoogle Scholar
  2. 2.
    Leo, M., Medioni, G., Trivedi, M., Kanade, T., Farinella, G.M.: Computer vision for assistive technologies. Comput. Vis. Image Underst. 154, 1–15 (2017)CrossRefGoogle Scholar
  3. 3.
    Razani, J., Wong, J.T., Dafaeeboini, N., et al.: Predicting everyday functional abilities of dementia patients with the mini mental state exam. J. Geriatr. Psychiatry Neurol. 22(1), 62–70 (2009)CrossRefGoogle Scholar
  4. 4.
    Vincze, M., Bajones, M., Suchi, M., Wolf, D., Weiss, A., Fischinger, D., da la Puente, P.: Learning and detecting objects with a mobile robot to assist older adults in their homes. In: Hua, G., Jégou, H. (eds.) ECCV 2016. LNCS, vol. 9914, pp. 316–330. Springer, Cham (2016).  https://doi.org/10.1007/978-3-319-48881-3_22 CrossRefGoogle Scholar
  5. 5.
    Yamazaki, K., Ueda, R., Nozawa, S., Kojima, M., Okada, K., Matsumoto, K., Ishikawa, M., Shimoyama, I., Inaba, M.: Home-assistant robot for an aging society. Proc. IEEE, Centennial Year, Special Issue, Quality Life Technol. 100(8), 2429–2441 (2012)Google Scholar
  6. 6.
    Kanade, T., Hebert, M.: First-person vision. Proc. IEEE 100(8), 2442–2453 (2012)CrossRefGoogle Scholar
  7. 7.
    Damen, D., Leelasawassuk, T., Mayol-Cuevas, W.: You-Do, I-Learn: egocentric unsupervised discovery of objects and their modes of interaction towards video-based guidance. Comput. Vis. Image Underst. (CVIU) 149, 98–112 (2016)CrossRefGoogle Scholar
  8. 8.
    Soran, B., Farhadi, A., Shapiro, L.: Generating notifications for missing actions: don’t forget to turn the lights off! In: IEEE International Conference on Computer Vision (2015)Google Scholar
  9. 9.
    Furnari, A., Farinella, G.M., Battiato, S.: Recognizing personal locations from egocentric videos. IEEE Trans. Human-Mach. Syst. 47, 6–18 (2017)Google Scholar
  10. 10.
    Ortis, A., Farinella, G.M., Damico, D., Addesso, L., Torrisi, G., Battiato, S.: Organizing egocentric videos for daily living monitoring. In: Lifelogging Tools and Applications in Conjunction with ACM Multimedia, Amsterdam (2016)Google Scholar
  11. 11.
    Ragusa, F., Tomaselli, V., Furnari, A., Battiato, S., Farinella, G.M.: Food vs non-food classification. In: 2nd International Workshop on Multimedia Assisted Dietary Management (MADiMa) in Conjunction with ACM Multimedia, Amsterdam (2016)Google Scholar
  12. 12.
    Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classication with deep convolutional neural networks. In: Advances in Neural Information Processing Systems (2012)Google Scholar
  13. 13.
    Berkeley Vision and Learning Center (BVLC). Cae, http://caffe.berkeleyvision.org/
  14. 14.
    Chih-Chung, C., Chih-Jen, L.: Libsvm: a library for support vector machines (2001)Google Scholar
  15. 15.
    Singla, A., Yuan, L., Ebrahimi, T.: Food/non-food image classification and food categorization using pre-trained GoogLeNet model. In: Proceedings of the 2nd International Workshop on Multimedia Assisted Dietary Management (2016)Google Scholar
  16. 16.
    Farinella, G.M., Allegra, D., Stanco, F., Battiato, S.: On the exploitation of one class classification to distinguish food vs non-food images. In: Murino, V., Puppo, E., Sona, D., Cristani, M., Sansone, C. (eds.) ICIAP 2015. LNCS, vol. 9281, pp. 375–383. Springer, Cham (2015).  https://doi.org/10.1007/978-3-319-23222-5_46 CrossRefGoogle Scholar
  17. 17.
    Kagaya, H., Aizawa, K., Ogawa, M.: Food detection and recognition using convolutional neural network. In: ACM International Conference on Multimedia, pp. 1085–1088 (2014)Google Scholar
  18. 18.
    Shimoda, W., Yanai, K.: Foodness proposal for multiple food detection by training of single food images. In: International Workshop on Multimedia Assisted Dietary Management, pp. 13–21 (2016)Google Scholar
  19. 19.
    Merler, M., Wu, H., Uceda-Sosa, R., Nguyen, Q.B., Smith, J.R.: Snap, eat, RepEat: a food recognition engine for dietary logging. In: Proceedings of the 2nd International Workshop on Multimedia Assisted Dietary Management, pp. 31–40 (2016)Google Scholar
  20. 20.
    Farinella, G.M., Moltisanti, M., Battiato, S.: Classifying food images represented as bag of Textons. In: IEEE International Conference on Image Processing (ICIP), Paris, pp. 5212–5216 (2014)Google Scholar
  21. 21.
    Martinel, N., Piciarelli, C., Micheloni, C.: A supervised extreme learning committee for food recognition journal article. Comput. Vis. Image Underst. 148, 67–86 (2016)CrossRefGoogle Scholar
  22. 22.
    Dehais, J., Anthimopoulos, M., Mougiakakou, S.: Food image segmentation for dietary assessment. In: International Workshop on Multimedia Assisted Dietary Management, pp. 23–28 (2016)Google Scholar
  23. 23.
    Shimoda, W., Yanai, K.: CNN-based food image segmentation without pixel-wise annotation. In: Murino, V., Puppo, E., Sona, D., Cristani, M., Sansone, C. (eds.) ICIAP 2015. LNCS, vol. 9281, pp. 449–457. Springer, Cham (2015).  https://doi.org/10.1007/978-3-319-23222-5_55 CrossRefGoogle Scholar
  24. 24.
    Ciocca, G., Napoletano, P., Schettini, R.: Food recognition and leftover estimation for daily diet monitoring. In: Murino, V., Puppo, E., Sona, D., Cristani, M., Sansone, C. (eds.) ICIAP 2015. LNCS, vol. 9281, pp. 334–341. Springer, Cham (2015).  https://doi.org/10.1007/978-3-319-23222-5_41 CrossRefGoogle Scholar
  25. 25.
    Ciocca, G., Napoletano, P., Schettini, R.: Food recognition: a new dataset, experiments and results. IEEE J. Biomed. Health Inf. 21(3), 588–598 (2017)CrossRefGoogle Scholar
  26. 26.
    Myers, A., Johnston, N., Rathod, V., Korattikara, A., Gorban, A., Silberman, N., Guadarrama, S., Papandreou, G., Huang, J., Murphy, K.: Im2Calories: towards an automated mobile vision food diary. In: IEEE International Conference on Computer Vision, pp. 1233–1241 (2015)Google Scholar
  27. 27.
    Beijbom, O., Joshi, N., Morris, D., Saponas, S., Khullar, S.: Menu-match: restaurant-specific food logging from images. In: IEEE Winter Conference on Applications of Computer Vision, pp. 844–851 (2015)Google Scholar
  28. 28.
    Bossard, L., Guillaumin, M., Van Gool, L.: Food-101 – mining discriminative components with random forests. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8694, pp. 446–461. Springer, Cham (2014).  https://doi.org/10.1007/978-3-319-10599-4_29 Google Scholar
  29. 29.
    LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521(7553), 436–444 (2015)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing AG 2017

Authors and Affiliations

  1. 1.Department of Mathematics and Computer Science, Image Processing LaboratoryUniversity of CataniaCataniaItaly

Personalised recommendations