Food Recognition and Dietary Assessment for Healthcare System at Mobile Device End Using Mask R-CNN

  • Hui Ye
  • Qiming ZouEmail author
Conference paper
Part of the Lecture Notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering book series (LNICST, volume 309)


Monitoring and estimation of food intake is of great significance to health-related research, such as obesity management. Traditional dietary records are performed in manual way. These methods are of low efficiency and a waste of labor, which are highly dependent on human interaction. In recent years, some researches have made progress in the estimation of food intake by using the computer vision technology. However, the recognition results of these researches are usually for the whole food object in the image, and the accuracy is not high. In terms of this problem, we provide a method to the food smart recognition and automatic dietary assessment on the mobile device. First, the food image is processed by MASK R-CNN which is more efficient than traditional methods. And more accurate recognition, classification and segmentation results of the multiple food items are output. Second, the OpenCV is used to display the food category and the corresponding food information of unit volume on the recognition page. Finally, in order to facilitate daily use, TensorFlow Lite is used to process the model to transplant to the mobile device, which can help to monitor people’s dietary intake.


Food image processing Dietary monitoring Mobile terminal recognition 


  1. 1.
  2. 2.
    He, K., Gkioxari, G., Dollar, P., et al.: Mask R-CNN. IEEE Trans. Pattern Anal. Mach. Intell. PP(99), 1 (2017)Google Scholar
  3. 3.
    Parrish, E., Goksel, A.K.: Pictorial pattern recognition applied to fruit harvesting. Trans. ASAE 20, 822–827 (1977)CrossRefGoogle Scholar
  4. 4.
    Yang, S., Chen, M., Pomerleau, D., et al.: Food recognition using statistics of pairwise local features. In: The Twenty-Third IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2010, San Francisco, CA, USA, 13–18 June 2010. IEEE (2010)Google Scholar
  5. 5.
    Matsuda, Y., Hoashi, H., Yanai, K.: Recognition of multiple-food images by detecting candidate regions. In: 2012 IEEE International Conference on Multimedia and Expo (ICME). IEEE (2012)Google Scholar
  6. 6.
    Martinel, N., Foresti, G.L., Micheloni, C.: Wide-slice residual networks for food recognition. In: Proceedings of 2018 IEEE Winter Conference on Applications of Computer Vision, WACV 2018, vol. 2018-January, pp. 567–576, December 2016Google Scholar
  7. 7.
    He, H., Kong, F., Tan, J.: DietCam: multiview food recognition using a multikernel SVM. IEEE J. Biomed. Health Inf. 20(3), 848–855 (2017)CrossRefGoogle Scholar
  8. 8.
    Williamson, D.A., Allen, H.R.: Digital photography: a new method for estimating food intake in cafeteria settings. Eat. Weight Disord. – Stud. Anorexia Bulimia Obes. 9(1), 24–28 (2004)CrossRefGoogle Scholar
  9. 9.
    Noronha, J., Hysen, E., Zhang, H., Gajos, K.Z.: Platemate. In: Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology - UIST 2011, p. 1 (2011)Google Scholar
  10. 10.
    Puri, M., Zhu, Z., Yu, Q., et al.: Recognition and volume estimation of food intake using a mobile device. In: IEEE Workshop on Applications of Computer Vision (WACV 2009), Snowbird, UT, USA, 7–8 December 2009. IEEE (2009)Google Scholar
  11. 11.
    Joutou, T., Yanai, K.: A food image recognition system with multiple kernel learning. In: IEEE International Conference on Image Processing. IEEE Press (2009)Google Scholar
  12. 12.
    Kong, F., Tan, J.: DietCam: automatic dietary assessment with mobile camera phones. Pervasive Mob. Comput. 8(1), 147–163 (2012)CrossRefGoogle Scholar
  13. 13.
    Kawano, Y., Yanai, K.: Real-time mobile food recognition system. In: 2013 IEEE Conference on Computer Vision and Pattern Recognition Workshops. IEEE (2013)Google Scholar
  14. 14.
    Kawano, Y., Yanai, K.: FoodCam: a real-time food recognition system on a smartphone. Multimed. Tools Appl. 74(14), 5263–5287 (2015)CrossRefGoogle Scholar
  15. 15.
    Pouladzadeh, P., Shirmohammadi, S., Arici, T.: Intelligent SVM based food intake measurement system. In: IEEE International Conference on Computational Intelligence & Virtual Environments for Measurement Systems & Applications. IEEE (2013)Google Scholar
  16. 16.
    Boohee Homepage. Accessed 20 Oct 2019
  17. 17.
    COCO dataset Homepage. Accessed 20 Oct 2019

Copyright information

© ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering 2020

Authors and Affiliations

  1. 1.School of Computer Engineering and ScienceShanghai UniversityShanghaiChina
  2. 2.Computing CenterShanghai UniversityShanghaiChina

Personalised recommendations