Food Recognition for Dietary Assessment Using Deep Convolutional Neural Networks

  • Stergios ChristodoulidisEmail author
  • Marios Anthimopoulos
  • Stavroula Mougiakakou
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9281)


Diet management is a key factor for the prevention and treatment of diet-related chronic diseases. Computer vision systems aim to provide automated food intake assessment using meal images. We propose a method for the recognition of already segmented food items in meal images. The method uses a 6-layer deep convolutional neural network to classify food image patches. For each food item, overlapping patches are extracted and classified and the class with the majority of votes is assigned to it. Experiments on a manually annotated dataset with 573 food items justified the choice of the involved components and proved the effectiveness of the proposed system yielding an overall accuracy of 84.9%.


Food recognition Convolutional neural networks Dietary management Machine learning 


  1. 1.
    Chen, M., Dhingra, K., Wu, W., Yang, L., Sukthankar, R.,Yang, J.: PFID: Pittsburgh fast-food image dataset. In: 16th IEEE International Conference on Image Processing (2009)Google Scholar
  2. 2.
    Anthimopoulos, M., Dehais, J., Diem, P., Mougiakakou, S.: Segmentation and recognition of multi-food meal images for carbohydrate counting. In: IEEE BIBE (2013)Google Scholar
  3. 3.
    Aizawa, K., Maruyama, Y., He, L., Morikawa, C.: Food Balance Estimation by Using Personal Dietary Tendencies in a Multimedia Food Log. IEEE Transactions on Multimedia 15(8), 2176–2185 (2013)CrossRefGoogle Scholar
  4. 4.
    Fengqing, Z., Bosch, M., Khanna, N., Boushey, C.J., Delp, E.J.: Multiple Hypotheses Image Segmentation and Classification With Application to Dietary Assessment. IEEE Journal of Biomedical and Health Informatics 19(1), 377–388 (2015)CrossRefGoogle Scholar
  5. 5.
    Oliveira, L., Costa, V., Neves, G., Oliveira, T., Jorge, E., Lizarraga, M.: A mobile, lightweight, poll-based food identification system. Pattern Recognition 47, 1941–1952 (2014)CrossRefGoogle Scholar
  6. 6.
    Chen, M.Y., Yang, Y.H., Ho, C.J., Wang, S.H., Liu, S.M., Chang, E., Yeh, C.H., Ouhyoung, M.: Automatic chinese food identification and quantity estimation. In: SIGGRAPH Asia 2012 (2012)Google Scholar
  7. 7.
    Matsuda, Y., Hoashi, H., Yanai, K.: Recognition of multiple-food images by detecting candidate regions. In: IEEE International Conference on Multimedia and Expo (2012)Google Scholar
  8. 8.
    Puri, M., Zhu, Z., Yu, Q., Divakaran, A., Sawhney, H.: Recognition and volume estimation of food intake using a mobile device. In: IEEE WACV, pp. 1–8 (2009)Google Scholar
  9. 9.
    Anthimopoulos, M.M., Gianola, L., Scarnato, L., Diem, P., Mougiakakou, S.G.: A Food Recognition System for Diabetic Patients Based on an Optimized Bag-of-Features Model. IEEE Journal of Biomedical and Health Informatics 18(4), 1261–1271 (2014)CrossRefGoogle Scholar
  10. 10.
    Bettadapura, V., Thomaz, E., Parnami, A., Abowd, G.D., Essa, I.A.: Leveraging context to support automated food recognition in restaurants. In: WACV 2015, pp. 580−587 (2015)Google Scholar
  11. 11.
    Kawano, Y., Yanai, K.: Food image recognition with deep convolutional features. In: ACM UbiComp Workshop on Cooking and Eating Activities (CEA) (2014)Google Scholar
  12. 12.
    Beijbom, O., Joshi, N., Morris, D., Saponas, S., Khullar, S.: Menu-match: restaurant-specific food logging from images. In: 2015 IEEE Winter Conference on Applications of Computer Vision, pp. 844−851 (2015)Google Scholar
  13. 13.
    Kawano, Y., Yanai, K.: FoodCam: A real-time food recognition system on a smartphone. Multimedia Tools and Applications (2014)Google Scholar
  14. 14.
    Farinella, G.M., Moltisanti, M., Battiato, S.: Classifying food images represented as bag of textons. In: IEEE International Conference on Image Processing (ICIP), pp. 5212−5216 (2014)Google Scholar
  15. 15.
    Yang, S., Chen, M., Pomerleau, D., Sukthankar, R.: Food recognition using statistics of pairwise local features. In: CVPR 2010 (2010)Google Scholar
  16. 16.
    Nguyen, D.T., Zong, Z., Ogunbona, P., Probst, Y.C., Li, W.: Food image classification using local appearance and global structural information. Neurocomputing 140, 242–251 (2014)CrossRefGoogle Scholar
  17. 17.
    Bossard, L., Guillaumin, M., Van Gool, L.: Food-101 – Mining discriminative components with random forests. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) ECCV 2014, Part VI. LNCS, vol. 8694, pp. 446–461. Springer, Heidelberg (2014)Google Scholar
  18. 18.
    Krizhevsky, A., Sutskever, I., Hinton, G.: ImageNet classification with deep convolutional neural networks. In: NIPS 2012 (2012)Google Scholar
  19. 19.
    Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet Large Scale Visual Recognition Challenge (2014)Google Scholar
  20. 20.
    Kagaya, H., Aizawa, K., Ogawa, K.: Food Detection and Recognition Using Convolutional Neural Network. ACM Multimedia, 1085−1088 (2014)Google Scholar
  21. 21.
    Hinton, G., Srivastava, N., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Improving neural networks by preventing co-adaptation of feature detectors.
  22. 22.
    Yangqing, J.: Caffe: An open source convolutional architecture for fast feature embedding (2013).

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Stergios Christodoulidis
    • 1
    • 2
    Email author
  • Marios Anthimopoulos
    • 1
    • 3
  • Stavroula Mougiakakou
    • 1
    • 4
  1. 1.ARTORG Center for Biomedical Engineering ResearchUniversity of BernBernSwitzerland
  2. 2.Graduate School of Cellular and Biomedical SciencesUniversity of BernBernSwitzerland
  3. 3.Department of Emergency MedicineBern University HospitalBernSwitzerland
  4. 4.Department of Endocrinology, Diabetes and Clinical NutritionBern University HospitalBernSwitzerland

Personalised recommendations