Skip to main content
SpringerLink
Log in

S2ML-TL Framework for Multi-label Food Recognition

  • Conference paper
  • First Online:
Pattern Recognition. ICPR International Workshops and Challenges (ICPR 2021)

Part of the book series: Lecture Notes in Computer Science ((LNIP,volume 12665))

Included in the following conference series:

Abstract

Transfer learning can be attributed to several recent breakthroughs in deep learning. It has shown upbeat performance improvements, but most of the transfer learning applications are confined towards fine-tuning. Transfer learning facilitates the learnability of the networks on domains with less data. However, learning becomes a difficult task with complex domains, such as multi-label food recognition, owing to the shear number of food classes as well as to the fine-grained nature of food images. For this purpose, we propose S2ML-TL, a new transfer learning framework to leverage the knowledge learnt on a simpler single-label food recognition task onto multi-label food recognition. The framework is further enhanced using class priors to tackle the dataset bias that exists between single-label and multi-label food domains. We validate the proposed scheme with two multi-label datasets on different backbone architectures and the results show improved performance compared to the conventional transfer learning approach.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 84.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 109.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Adel, T., Zhao, H., Wong, A.: Unsupervised domain adaptation with a relaxed covariate shift assumption. In: Thirty-First AAAI Conference on Artificial Intelligence (2017)

    Google Scholar 

  2. Aguilar, E., Bolaños, M., Radeva, P.: Food recognition using fusion of classifiers based on CNNs. In: Battiato, S., Gallo, G., Schettini, R., Stanco, F. (eds.) ICIAP 2017. LNCS, vol. 10485, pp. 213–224. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-68548-9_20

    Chapter  Google Scholar 

  3. Aguilar, E., Bolaños, M., Radeva, P.: Regularized uncertainty-based multi-task learning model for food analysis. J. Vis. Commun. Image Represent. 60, 360–370 (2019)

    Article  Google Scholar 

  4. Aguilar, E., Radeva, P.: Uncertainty-aware integration of local and flat classifiers for food recognition. Pattern Recogn. Lett. 136, 237–243 (2020)

    Article  Google Scholar 

  5. Aguilar, E., Remeseiro, B., Bolaños, M., Radeva, P.: Grab, pay, and eat: semantic food detection for smart restaurants. IEEE Trans. Multimedia 20(12), 3266–3275 (2018)

    Article  Google Scholar 

  6. Anzawa, M., Amano, S., Yamakata, Y., Motonaga, K., Kamei, A., Aizawa, K.: Recognition of multiple food items in a single photo for use in a buffet-style restaurant. IEICE Trans. Inf. Syst. 102(2), 410–414 (2019)

    Article  Google Scholar 

  7. Bolaños, M., Ferrà, A., Radeva, P.: Food ingredients recognition through multi-label learning. In: Battiato, S., Farinella, G.M., Leo, M., Gallo, G. (eds.) ICIAP 2017. LNCS, vol. 10590, pp. 394–402. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-70742-6_37

    Chapter  Google Scholar 

  8. Bossard, L., Guillaumin, M., Van Gool, L.: Food-101 – mining discriminative components with random forests. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014. LNCS, vol. 8694, pp. 446–461. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10599-4_29

    Chapter  Google Scholar 

  9. Byrd, J., Lipton, Z.: What is the effect of importance weighting in deep learning? In: International Conference on Machine Learning, pp. 872–881 (2019)

    Google Scholar 

  10. Cao, Z., Long, M., Wang, J., Jordan, M.I.: Partial transfer learning with selective adversarial networks. In: CVPR, pp. 2724–2732 (2018)

    Google Scholar 

  11. Chen, J., Ngo, C.W.: Deep-based ingredient recognition for cooking recipe retrieval. In: Proceedings of the 24th ACM International Conference on Multimedia, pp. 32–41 (2016)

    Google Scholar 

  12. Chen, X., Zhou, H., Zhu, Y., Diao, L.: ChineseFoodNet: a large-scale image dataset for Chinese food recognition. arXiv preprint arXiv:1705.02743 (2017)

  13. Ciocca, G., Napoletano, P., Schettini, R.: Food recognition: a new dataset, experiments, and results. IEEE J. Biomed. Health Inform. 21(3), 588–598 (2016)

    Article  Google Scholar 

  14. Donadello, I., Dragoni, M.: Ontology-driven food category classification in images. In: Ricci, E., Rota Bulò, S., Snoek, C., Lanz, O., Messelodi, S., Sebe, N. (eds.) ICIAP 2019. LNCS, vol. 11752, pp. 607–617. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-30645-8_55

    Chapter  Google Scholar 

  15. El Khoury, C.F., Karavetian, M., Halfens, R.J., Crutzen, R., Khoja, L., Schols, J.M.: The effects of dietary mobile apps on nutritional outcomes in adults with chronic diseases: a systematic review and meta-analysis. J. Acad. Nutr. Diet. 119(4), 626–651 (2019)

    Article  Google Scholar 

  16. Godasu, R., El-Gayar, O., Sutrave, K.: Multi-stage transfer learning system with light-weight architectures in medical image classification (2020)

    Google Scholar 

  17. Grover, A., et al.: Bias correction of learned generative models using likelihood-free importance weighting. In: NIPS, pp. 11058–11070 (2019)

    Google Scholar 

  18. Hassannejad, H., Matrella, G., Ciampolini, P., De Munari, I., Mordonini, M., Cagnoni, S.: Food image recognition using very deep convolutional networks. In: Proceedings of the 2nd International Workshop on Multimedia Assisted Dietary Management, pp. 41–49 (2016)

    Google Scholar 

  19. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. corr abs/1512.03385 (2015)

    Google Scholar 

  20. Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)

  21. Horiguchi, S., Amano, S., Ogawa, M., Aizawa, K.: Personalized classifier for food image recognition. IEEE Trans. Multimedia 20(10), 2836–2848 (2018)

    Article  Google Scholar 

  22. Jiang, L., Qiu, B., Liu, X., Huang, C., Lin, K.: DeepFood: food image analysis and dietary assessment via deep model. IEEE Access 8, 47477–47489 (2020)

    Article  Google Scholar 

  23. Kagaya, H., Aizawa, K., Ogawa, M.: Food detection and recognition using convolutional neural network. In: Proceedings of the 22nd ACM International Conference on Multimedia, pp. 1085–1088 (2014)

    Google Scholar 

  24. Kaur, P., Sikka, K., Wang, W., Belongie, S., Divakaran, A.: FoodX-251: a dataset for fine-grained food classification. arXiv preprint arXiv:1907.06167 (2019)

  25. Kawano, Y., Yanai, K.: Automatic expansion of a food image dataset leveraging existing categories with domain adaptation. In: Agapito, L., Bronstein, M.M., Rother, C. (eds.) ECCV 2014. LNCS, vol. 8927, pp. 3–17. Springer, Cham (2015). https://doi.org/10.1007/978-3-319-16199-0_1

    Chapter  Google Scholar 

  26. Kimura, M., Izawa, R.: Density fixing: simple yet effective regularization method based on the class prior. arXiv preprint arXiv:2007.03899 (2020)

  27. Lee, K.H., He, X., Zhang, L., Yang, L.: CleanNet: transfer learning for scalable image classifier training with label noise. In: CVPR, pp. 5447–5456 (2018)

    Google Scholar 

  28. Liu, C., Cao, Yu., Luo, Y., Chen, G., Vokkarane, V., Ma, Y.: DeepFood: deep learning-based food image recognition for computer-aided dietary assessment. In: Chang, C.K., Chiari, L., Cao, Yu., Jin, H., Mokhtari, M., Aloulou, H. (eds.) ICOST 2016. LNCS, vol. 9677, pp. 37–48. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-39601-9_4

    Chapter  Google Scholar 

  29. MacKay, D.J., Mac Kay, D.J.: Information Theory, Inference and Learning Algorithms. Cambridge University Press, Cambridge (2003)

    Google Scholar 

  30. Malinin, A., Gales, M.: Predictive uncertainty estimation via prior networks. In: Advances in Neural Information Processing Systems, pp. 7047–7058 (2018)

    Google Scholar 

  31. Martinel, N., Foresti, G.L., Micheloni, C.: Wide-slice residual networks for food recognition. In: 2018 IEEE WACV, pp. 567–576. IEEE (2018)

    Google Scholar 

  32. Matsuda, Y., Yanai, K.: Multiple-food recognition considering co-occurrence employing manifold ranking. In: Proceedings of the 21st International Conference on Pattern Recognition (ICPR 2012), pp. 2017–2020. IEEE (2012)

    Google Scholar 

  33. Meyers, A., et al.: Im2Calories: towards an automated mobile vision food diary. In: ICCV, pp. 1233–1241 (2015)

    Google Scholar 

  34. Mezgec, S., Seljak, B.K.: Using deep learning for food and beverage image recognition. In: 2019 IEEE International Conference on Big Data (Big Data), pp. 5149–5151. IEEE (2019)

    Google Scholar 

  35. Miasnikov, E., Savchenko, A.: Detection and recognition of food in photo galleries for analysis of user preferences. In: Campilho, A., Karray, F., Wang, Z. (eds.) ICIAR 2020. LNCS, vol. 12131, pp. 83–94. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-50347-5_9

    Chapter  Google Scholar 

  36. Min, W., Jiang, S., Liu, L., Rui, Y., Jain, R.: A survey on food computing. ACM Comput. Surv. (CSUR) 52(5), 1–36 (2019)

    Article  Google Scholar 

  37. Min, W., Liu, L., Luo, Z., Jiang, S.: Ingredient-guided cascaded multi-a ention network for food recognition (2019)

    Google Scholar 

  38. Min, W., et al.: ISIA Food-500: a dataset for large-scale food recognition via stacked global-local attention network. arXiv preprint arXiv:2008.05655 (2020)

  39. Ming, Z.-Y., Chen, J., Cao, Yu., Forde, C., Ngo, C.-W., Chua, T.S.: Food photo recognition for dietary tracking: system and experiment. In: Schoeffmann, K., et al. (eds.) MMM 2018. LNCS, vol. 10705, pp. 129–141. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-73600-6_12

    Chapter  Google Scholar 

  40. Nag, N., Pandey, V., Jain, R.: Health multimedia: lifestyle recommendations based on diverse observations. In: Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval, pp. 99–106 (2017)

    Google Scholar 

  41. Ng, H.W., Nguyen, V.D., Vonikakis, V., Winkler, S.: Deep learning for emotion recognition on small datasets using transfer learning. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, pp. 443–449 (2015)

    Google Scholar 

  42. Oka, R., et al.: Study protocol for the effects of artificial intelligence (AI)-supported automated nutritional intervention on glycemic control in patients with type 2 diabetes mellitus. Diabetes Ther. 10(3), 1151–1161 (2019)

    Article  Google Scholar 

  43. World Health Organization, et al.: Healthy diet. Technical report, World Health Organization. Regional Office for the Eastern Mediterranean (2019)

    Google Scholar 

  44. Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2009)

    Article  Google Scholar 

  45. Pandey, P., Deepthi, A., Mandal, B., Puhan, N.B.: FoodNet: recognizing foods using ensemble of deep networks. IEEE Sig. Process. Lett. 24(12), 1758–1762 (2017)

    Article  Google Scholar 

  46. Peng, Z., Zhang, W., Han, N., Fang, X., Kang, P., Teng, L.: Active transfer learning. IEEE Trans. Circ. Syst. Video Tech. 30(4), 1022–1036 (2019)

    Google Scholar 

  47. Sakai, T., Shimizu, N.: Covariate shift adaptation on learning from positive and unlabeled data. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 4838–4845 (2019)

    Google Scholar 

  48. Selamat, N.A., Ali, S.H.M.: Automatic food intake monitoring based on chewing activity: a survey. IEEE Access 8, 48846–48869 (2020)

    Article  Google Scholar 

  49. Joachims, T.: Text categorization with support vector machines: learning with many relevant features. In: Nédellec, C., Rouveirol, C. (eds.) ECML 1998. LNCS, vol. 1398, pp. 137–142. Springer, Heidelberg (1998). https://doi.org/10.1007/BFb0026683

    Chapter  Google Scholar 

  50. Shimoda, W., Yanai, K.: Foodness proposal for multiple food detection by training of single food images. In: Proceedings of the 2nd International Workshop on Multimedia Assisted Dietary Management, pp. 13–21 (2016)

    Google Scholar 

  51. Stojanov, P., Gong, M., Carbonell, J.G., Zhang, K.: Low-dimensional density ratio estimation for covariate shift correction. In: Proceedings of Machine Learning Research, vol. 89, p. 3449 (2019)

    Google Scholar 

  52. Szegedy, C., Ioffe, S., Vanhoucke, V., Alemi, A.A.: Inception-v4, inception-ResNet and the impact of residual connections on learning. In: Thirty-First AAAI Conference on Artificial Intelligence (2017)

    Google Scholar 

  53. Tahir, G.A., Loo, C.K.: An open-ended continual learning for food recognition using class incremental extreme learning machines. IEEE Access 8, 82328–82346 (2020)

    Article  Google Scholar 

  54. Tan, C., Sun, F., Kong, T., Zhang, W., Yang, C., Liu, C.: A survey on deep transfer learning. In: Kůrková, V., Manolopoulos, Y., Hammer, B., Iliadis, L., Maglogiannis, I. (eds.) ICANN 2018. LNCS, vol. 11141, pp. 270–279. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01424-7_27

    Chapter  Google Scholar 

  55. Tanno, R., Okamoto, K., Yanai, K.: DeepFoodCam: a DCNN-based real-time mobile food recognition system. In: Proceedings of the 2nd International Workshop on Multimedia Assisted Dietary Management, pp. 89–89 (2016)

    Google Scholar 

  56. Tasci, E.: Voting combinations-based ensemble of fine-tuned convolutional neural networks for food image recognition. Multimed. Tools Appl. 79, 30397–30418 (2020). https://doi.org/10.1007/s11042-020-09486-1

    Article  Google Scholar 

  57. Torrey, L., Shavlik, J.: Transfer learning. In: Handbook of Research on Machine Learning Applications and Trends: Algorithms, Methods, and Techniques, pp. 242–264. IGI Global (2010)

    Google Scholar 

  58. Wang, M., Deng, W.: Deep visual domain adaptation: a survey. Neurocomputing 312, 135–153 (2018)

    Article  Google Scholar 

  59. Wang, X., Li, L., Ye, W., Long, M., Wang, J.: Transferable attention for domain adaptation. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 5345–5352 (2019)

    Google Scholar 

  60. Wang, Y., Chen, J.J., Ngo, C.W., Chua, T.S., Zuo, W., Ming, Z.: Mixed dish recognition through multi-label learning. In: Proceedings of the 11th Workshop on Multimedia for Cooking and Eating Activities, pp. 1–8 (2019)

    Google Scholar 

  61. Wang, Y., Chen, J.J., Ngo, C.W., Chua, T.S., Zuo, W., Ming, Z.: Mixed dish recognition through multi-label learning. In: Proceedings of the 11th Workshop on Multimedia for Cooking and Eating Activities, CEA 2019, p. 1–8. Association for Computing Machinery, New York (2019)

    Google Scholar 

  62. Wang, Z., Dai, Z., Póczos, B., Carbonell, J.: Characterizing and avoiding negative transfer. In: CVPR, pp. 11293–11302 (2019)

    Google Scholar 

  63. Wu, H., Merler, M., Uceda-Sosa, R., Smith, J.R.: Learning to make better mistakes: semantics-aware visual food recognition. In: Proceedings of the 24th ACM International Conference on Multimedia, pp. 172–176 (2016)

    Google Scholar 

  64. Wu, J., Huang, Z., Thoma, J., Acharya, D., Van Gool, L.: Wasserstein divergence for GANs. In: Proceedings of ECCV, pp. 653–668 (2018)

    Google Scholar 

  65. Yanai, K., Kawano, Y.: Food image recognition using deep convolutional network with pre-training and fine-tuning. In: 2015 IEEE ICMEW, pp. 1–6. IEEE (2015)

    Google Scholar 

  66. Zhang, Y., Parker, A.G.: Eat4Thought: a design of food journaling. In: Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1–8 (2020)

    Google Scholar 

  67. Zhao, H., Yap, K.H., Kot, A.C., Duan, L.: JDNet: a joint-learning distilled network for mobile visual food recognition. IEEE J. Sel. Top. Sig. Process. 14(4), 665–675 (2020)

    Article  Google Scholar 

  68. Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: CVPR, pp. 2921–2929 (2016)

    Google Scholar 

Download references

Acknowledgements

This work was partially funded by TIN2018-095232-B-C21, SGR-2017 1742, Nestore project of the European Commission Horizon 2020 programme (Grant no. 769643), Validithi EIT Health program and CERCA Programme/Generalitat de Catalunya. We acknowledge the support of NVIDIA Corporation with the donation of the Titan Xp GPUs.

Author information

Authors and Affiliations

  1. Dept. de Matemàtiques i Informàtica, Universitat de Barcelona, Barcelona, Spain

    Bhalaji Nagarajan, Eduardo Aguilar & Petia Radeva

  2. Departamento de Ingeniería de Sistemas y Computación, Universidad Católica del Norte, Antofagasta, Chile

    Eduardo Aguilar

  3. Computer Vision Center, Cerdanyola, Barcelona, Spain

    Petia Radeva

Authors
  1. Bhalaji Nagarajan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Eduardo Aguilar
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Petia Radeva
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Bhalaji Nagarajan .

Editor information

Editors and Affiliations

  1. Dipartimento di Ingegneria dell’Informazione, University of Firenze, Firenze, Italy

    Alberto Del Bimbo

  2. Dipartimento di Ingegneria “Enzo Ferrari”, Università di Modena e Reggio Emilia, Modena, Italy

    Rita Cucchiara

  3. Department of Computer Science, Boston University, Boston, MA, USA

    Stan Sclaroff

  4. Dipartimento di Matematica e Informatica, University of Catania, Catania, Italy

    Giovanni Maria Farinella

  5. Cloud & AI, JD.COM, Beijing, China

    Tao Mei

  6. Dipartimento di Ingegneria dell’Informazione, University of Firenze, Firenze, Italy

    Marco Bertini

  7. Computational Sciences Department, National Institute of Astrophysics, Optics and Electronics (INAOE), Tonantzintla, Puebla, Mexico

    Hugo Jair Escalante

  8. Dipartimento di Ingegneria “Enzo Ferrari”, Università di Modena e Reggio Emilia, Modena, Italy

    Roberto Vezzani

Rights and permissions

Reprints and permissions

Copyright information

© 2021 Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Nagarajan, B., Aguilar, E., Radeva, P. (2021). S2ML-TL Framework for Multi-label Food Recognition. In: Del Bimbo, A., et al. Pattern Recognition. ICPR International Workshops and Challenges. ICPR 2021. Lecture Notes in Computer Science(), vol 12665. Springer, Cham. https://doi.org/10.1007/978-3-030-68821-9_50

Download citation

  • DOI: https://doi.org/10.1007/978-3-030-68821-9_50

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-030-68820-2

  • Online ISBN: 978-3-030-68821-9

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Societies and partnerships

Search

Navigation