Skip to main content
SpringerLink
Log in

Uncertainty Modeling and Deep Learning Applied to Food Image Analysis

  • Conference paper
  • First Online:
Biomedical Engineering Systems and Technologies (BIOSTEC 2020)

Part of the book series: Communications in Computer and Information Science ((CCIS,volume 1400))

Abstract

Recognizing food images arises as a difficult image recognition task due to the high intra-class variance and low inter-class variance of food categories. Deep learning has been shown as a promising methodology to address such difficult problems as food image recognition that can be considered as a fine-grained object recognition problem. We argue that, in order to continue improving performance in this task, it is necessary to better understand what the model learns instead of considering it as a black box. In this paper, we show how uncertainty analysis can help us gain a better understanding of the model in the context of the food recognition. Furthermore, we take decisions to improve its performance based on this analysis and propose a new data augmentation approach considering sample-level uncertainty. The results of our method considering the evaluation on a public food dataset are very encouraging.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  • Aguilar, E., Bolaños, M., Radeva, P.: Regularized uncertainty-based multi-task learning model for food analysis. J. Vis. Commun. Image Represent. 60, 360–370 (2019)

    Article  Google Scholar 

  • Aguilar, E., Radeva, P.: Class-conditional data augmentation applied to image classification. In: Vento, M., Percannella, G. (eds.) CAIP 2019. LNCS, vol. 11679, pp. 182–192. Springer, Cham (2019a). https://doi.org/10.1007/978-3-030-29891-3_17

  • Aguilar, E., Radeva, P.: Food recognition by integrating local and flat classifiers. In: Morales, A., Fierrez, J., Sánchez, J.S., Ribeiro, B. (eds.) IbPRIA 2019. LNCS, vol. 11867, pp. 65–74. Springer, Cham (2019b). https://doi.org/10.1007/978-3-030-31332-6_6

  • Aguilar, E., Remeseiro, B., Bolaños, M., Radeva, P.: Grab, pay, and eat: semantic food detection for smart restaurants. IEEE Trans. Multimed. 20(12), 3266–3275 (2018)

    Article  Google Scholar 

  • Ali-Gombe, A., Elyan, E.: MFC-GAN: class-imbalanced dataset classification using multiple fake class generative adversarial network. Neurocomputing 361, 212–221 (2019)

    Article  Google Scholar 

  • Alliance, I.U.N.: National adult nutrition survey. Public Health (2019)

    Google Scholar 

  • Anthimopoulos, M.M., Gianola, L., Scarnato, L., Diem, P., Mougiakakou, S.G.: A food recognition system for diabetic patients based on an optimized bag-of-features model. IEEE J. Biomed. Health Inform. 18(4), 1261–1271 (2014)

    Article  Google Scholar 

  • Anzawa, M., Amano, S., Yamakata, Y., Motonaga, K., Kamei, A., Aizawa, K.: Recognition of multiple food items in a single photo for use in a buffet-style restaurant. IEICE Trans. Inf. Syst. 102(2), 410–414 (2019)

    Article  Google Scholar 

  • Blundell, C., Cornebise, J., Kavukcuoglu, K., Wierstra, D.: Weight uncertainty in neural network. In: ICML, pp. 1613–1622 (2015)

    Google Scholar 

  • Bosch, M., Zhu, F., Khanna, N., Boushey, C.J., Delp, E.J.: Combining global and local features for food identification in dietary assessment. In: 2011 18th IEEE International Conference on Image Processing, pp. 1789–1792. IEEE (2011)

    Google Scholar 

  • Bossard, L., Guillaumin, M., Van Gool, L.: Food-101 – mining discriminative components with random forests. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8694, pp. 446–461. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10599-4_29

    Chapter  Google Scholar 

  • Bruno, V., Silva Resende, C.J.: A survey on automated food monitoring and dietary management systems. J. Health Med. Inform. 8(3) (2017)

    Google Scholar 

  • Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: DeepLab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Trans. Pattern Anal. Mach. Intell. 40(4), 834–848 (2017)

    Article  Google Scholar 

  • Chen, M., Dhingra, K., Wu, W., Yang, L., Sukthankar, R., Yang, J.: PFID: Pittsburgh fast-food image dataset. In: 2009 16th IEEE International Conference on Image Processing (ICIP), pp. 289–292. IEEE (2009)

    Google Scholar 

  • Ciocca, G., Napoletano, P., Schettini, R.: Food recognition: a new dataset, experiments and results. IEEE J. Biomed. Health Inform. 21(3), 588–598 (2017a)

    Google Scholar 

  • Ciocca, G., Napoletano, P., Schettini, R.: Learning CNN-based features for retrieval of food images. In: Battiato, S., Farinella, G.M., Leo, M., Gallo, G. (eds.) ICIAP 2017. LNCS, vol. 10590, pp. 426–434. Springer, Cham (2017b). https://doi.org/10.1007/978-3-319-70742-6_41

  • Donadello, I., Dragoni, M.: Ontology-driven food category classification in images. In: Ricci, E., Rota Bulò, S., Snoek, C., Lanz, O., Messelodi, S., Sebe, N. (eds.) ICIAP 2019. LNCS, vol. 11752, pp. 607–617. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-30645-8_55

    Chapter  Google Scholar 

  • El Khoury, C.F., Karavetian, M., Halfens, R.J., Crutzen, R., Khoja, L., Schols, J.M.: The effects of dietary mobile apps on nutritional outcomes in adults with chronic diseases: a systematic review. J. Acad. Nutr. Diet. 119, 626–651 (2019)

    Article  Google Scholar 

  • Gal, Y., Ghahramani, Z.: Dropout as a Bayesian approximation: representing model uncertainty in deep learning. In: ICML, pp. 1050–1059 (2016)

    Google Scholar 

  • Goodfellow, I., Mirza, M., Courville, A., Bengio, Y.: Multi-prediction deep Boltzmann machines. In: Advances in Neural Information Processing Systems, pp. 548–556 (2013)

    Google Scholar 

  • Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)

    Google Scholar 

  • Hassannejad, H., Matrella, G., Ciampolini, P., De Munari, I., Mordonini, M., Cagnoni, S.: Food image recognition using very deep convolutional networks. In: Proceedings of the 2nd International Workshop on MADiMa, pp. 41–49. ACM (2016)

    Google Scholar 

  • He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

    Google Scholar 

  • Chen, J., Ngo, C.W.: Deep-based ingredient recognition for cooking recipe retrival. In: ACM Multimedia (2016)

    Google Scholar 

  • Joutou, T., Yanai, K.: A food image recognition system with multiple kernel learning. In: 2009 16th IEEE International Conference on Image Processing (ICIP), pp. 285–288. IEEE (2009)

    Google Scholar 

  • Kaur, P., Sikka, K., Wang, W., Belongie, S., Divakaran, A.: Foodx-251: a dataset for fine-grained food classification. arXiv preprint arXiv:1907.06167 (2019)

  • Kendall, A., Gal, Y.: What uncertainties do we need in Bayesian deep learning for computer vision? In: Advances in Neural Information Processing Systems, pp. 5574–5584 (2017)

    Google Scholar 

  • Khan, S., Hayat, M., Zamir, S.W., Shen, J., Shao, L.: Striking the right balance with uncertainty. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 103–112 (2019)

    Google Scholar 

  • Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. In: Advances in Neural Information Processing Systems, pp. 1097–1105 (2012)

    Google Scholar 

  • Lee, K.-H., He, X., Zhang, L., Yang, L.: CleanNet: transfer learning for scalable image classifier training with label noise. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5447–5456 (2018)

    Google Scholar 

  • Liu, C., Cao, Yu., Luo, Y., Chen, G., Vokkarane, V., Ma, Y.: DeepFood: deep learning-based food image recognition for computer-aided dietary assessment. In: Chang, C.K., Chiari, L., Cao, Yu., Jin, H., Mokhtari, M., Aloulou, H. (eds.) ICOST 2016. LNCS, vol. 9677, pp. 37–48. Springer, Cham (2016a). https://doi.org/10.1007/978-3-319-39601-9_4

  • Liu, W., Wen, Y., Yu, Z., Yang, M.: Large-margin softmax loss for convolutional neural networks. In: ICML, vol. 2, p. 7 (2016b)

    Google Scholar 

  • Louizos, C., Welling, M.: Multiplicative normalizing flows for variational Bayesian neural networks. In: ICML vol. 70, pp. 2218–2227. JMLR.org (2017)

    Google Scholar 

  • Mariani, G., Scheidegger, F., Istrate, R., Bekas, C., Malossi, C.: Bagan: data augmentation with balancing GAN. arXiv preprint arXiv:1803.09655 (2018)

  • Martinel, N., Foresti, G.L., Micheloni, C.: Wide-slice residual networks for food recognition. In: 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 567–576. IEEE (2018)

    Google Scholar 

  • Matsuda, Y., Hoashi, H., Yanai, K.: Recognition of multiple-food images by detecting candidate regions. In: 2012 IEEE International Conference on Multimedia and Expo, pp. 25–30. IEEE (2012)

    Google Scholar 

  • Meyers, A., et al.: Im2calories: towards an automated mobile vision food diary. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1233–1241 (2015)

    Google Scholar 

  • Ming, Z.-Y., Chen, J., Cao, Yu., Forde, C., Ngo, C.-W., Chua, T.S.: Food photo recognition for dietary tracking: system and experiment. In: Schoeffmann, K., et al. (eds.) MMM 2018. LNCS, vol. 10705, pp. 129–141. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-73600-6_12

    Chapter  Google Scholar 

  • Mirza, M., Osindero, S.: Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 (2014)

  • Molchanov, D., Ashukha, A., Vetrov, D.: Variational dropout sparsifies deep neural networks. In: ICML, vol. 70, pp. 2498–2507. JMLR.org (2017)

    Google Scholar 

  • Nag, N., Pandey, V., Jain, R.: Health multimedia: lifestyle recommendations based on diverse observations. In: Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval, pp. 99–106. ACM (2017)

    Google Scholar 

  • Nielsen, C., Okoniewski, M.: GAN data augmentation through active learning inspired sample acquisition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 109–112 (2019)

    Google Scholar 

  • Odena, A., Olah, C., Shlens, J.: Conditional image synthesis with auxiliary classifier GANs. In: ICML, vol. 70, pp. 2642–2651. JMLR.org (2017)

    Google Scholar 

  • Sahoo, D., et al.: FoodAI: food image recognition via deep learning for smart food logging (2019)

    Google Scholar 

  • Sensoy, M., Kaplan, L., Kandemir, M.: Evidential deep learning to quantify classification uncertainty. In: Advances in Neural Information Processing Systems, pp. 3179–3189 (2018)

    Google Scholar 

  • Shaham, T.R., Dekel, T., Michaeli, T.: SinGAN: learning a generative model from a single natural image. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4570–4580 (2019)

    Google Scholar 

  • Subhi, M.A., Ali, S.H., Mohammed, M.A.: Vision-based approaches for automatic food recognition and dietary assessment: a survey. IEEE Access 7, 35370–35381 (2019)

    Article  Google Scholar 

  • Tanno, R., Okamoto, K., Yanai, K.: DeepFoodCam: a DCNN-based real-time mobile food recognition system. In: Proceedings of the 2nd International Workshop on MADiMa, p. 89. ACM (2016)

    Google Scholar 

  • Wang, Y., Chen, J., Ngo, C.-W., Chua, T.-S., Zuo, W., Ming, Z.: Mixed dish recognition through multi-label learning. In: Proceedings of the 11th Workshop on Multimedia for Cooking and Eating Activities, CEA 2019, pp. 1–80. Association for Computing Machinery, New York (2019)

    Google Scholar 

  • Wu, H., Merler, M., Uceda-Sosa, R., Smith, J.R.: Learning to make better mistakes: semantics-aware visual food recognition. In: Proceedings of the 24th ACM International Conference on Multimedia, pp. 172–176. ACM (2016)

    Google Scholar 

  • Yanai, K., Kawano, Y.: Food image recognition using deep convolutional network with pre-training and fine-tuning. In: 2015 IEEE International Conference on Multimedia And Expo Workshops (ICMEW), pp. 1–6. IEEE (2015)

    Google Scholar 

  • Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2921–2929 (2016)

    Google Scholar 

Download references

Acknowledgements

This work was partially funded by TIN2018-095232-B-C21, SGR-2017 1742, Nestore ID: 769643, Validithi and CERCA Programme/Generalitat de Catalunya. E. Aguilar acknowledges the support of CONICYT Becas Chile. P. Radeva is partially supported by ICREA Academia 2014. We acknowledge the support of NVIDIA Corporation with the donation of Titan Xp GPUs.

Author information

Authors and Affiliations

  1. Departamento de Ingeniería de Sistemas y Computación, Universidad Católica del Norte, Avenida Angamos 0610, Antofagasta, Chile

    Eduardo Aguilar

  2. Departament de Matemàtiques i Informàtica, Universitat de Barcelona, Gran Via de les Corts Catalanes 585, 08007, Barcelona, Spain

    Eduardo Aguilar, Bhalaji Nagarajan, Rupali Khatun, Marc Bolaños & Petia Radeva

  3. Computer Vision Center, Cerdanyola, Barcelona, Spain

    Petia Radeva

Authors
  1. Eduardo Aguilar
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bhalaji Nagarajan
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Rupali Khatun
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Marc Bolaños
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Petia Radeva
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Eduardo Aguilar .

Editor information

Editors and Affiliations

  1. Zhejiang University, Hangzhou, China

    Xuesong Ye

  2. Fraunhofer Portugal Research Center for Assistive Information and Communication Solutions, Porto, Portugal

    Filipe Soares

  3. Nice Sophia Antipolis University, Nice Cedex 2, France

    Elisabetta De Maria

  4. Universidad Politécnica de Madrid, Madrid, Spain

    Pedro Gómez Vilda

  5. University of Milano-Bicocca, Milan, Italy

    Federico Cabitza

  6. Instituto de Telecomunicações, University of Lisbon, Lisbon, Portugal

    Ana Fred

  7. Universidade Nova de Lisboa, Caparica, Portugal

    Hugo Gamboa

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Aguilar, E., Nagarajan, B., Khatun, R., Bolaños, M., Radeva, P. (2021). Uncertainty Modeling and Deep Learning Applied to Food Image Analysis. In: Ye, X., et al. Biomedical Engineering Systems and Technologies. BIOSTEC 2020. Communications in Computer and Information Science, vol 1400. Springer, Cham. https://doi.org/10.1007/978-3-030-72379-8_1

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-72379-8_1

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-72378-1

  • Online ISBN: 978-3-030-72379-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation