Dish Detection and Segmentation for Dietary Assessment on Smartphones

  • Joachim DehaisEmail author
  • Marios Anthimopoulos
  • Stavroula Mougiakakou
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9281)


Diet-related chronic diseases severely affect personal and global health. However, managing or treating these diseases currently requires long training and high personal involvement to succeed. Computer vision systems could assist with the assessment of diet by detecting and recognizing different foods and their portions in images. We propose novel methods for detecting a dish in an image and segmenting its contents with and without user interaction. All methods were evaluated on a database of over 1600 manually annotated images. The dish detection scored an average of 99% accuracy with a .2s/image run time, while the automatic and semi-automatic dish segmentation methods reached average accuracies of 88% and 91% respectively, with an average run time of .5s/image, outperforming competing solutions.


Diet assessment Diabetes Obesity Image segmentation Computer vision Smartphone 


  1. 1.
    Shroff, G, Smailagic, A., Siewiorek, D.P.: Wearable context-aware food recognition for calorie monitoring. In: 12th IEEE ISWC, pp. 119–120 (2008)Google Scholar
  2. 2.
    He, Y., Khanna, N., Boushey, C.J., Delp, E.J.: Image segmentation for image-based dietary assessment: a comparative study. In: IEEE ISSCS 2013Google Scholar
  3. 3.
    Kass, M., Witkin, A., Terzopoulos, D.: Snakes: Active contour models. Int. J. Comput. Vis. 1, 321–331 (1998)CrossRefGoogle Scholar
  4. 4.
    Shi, J., Malik, J.: Normalized cuts and image segmentation. IEEE Tran. Pattern Anal. Mach. Intell. 22(8), 888–905 (2000)CrossRefGoogle Scholar
  5. 5.
    Felzenszwalb, P.F., Huttenlocher, D.P.: Image segmentation using local variation. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 98–104 (1998)Google Scholar
  6. 6.
    Zhu, F., Bosch, M., Khanna, N., Boushey, C.J., Delp, E.J.: Multiple Hypotheses Image Segmentation and Classification With Application to Dietary Assessment. IEEE J. Biomed. Health Inform. 19(1), 377–388 (2015)CrossRefGoogle Scholar
  7. 7.
    Matsuda, Y., Hoashi, H., Yanai, K.: Recognition of multiple-food images by detecting candidate regions. In: IEEE ICME, 2012, pp. 25–30 (2012)Google Scholar
  8. 8.
    Felzenszwalb, P.F., Girshick, R.B., McAllester, D., Ramanan, D.: Object detection with discriminatively trained part-based models. IEEE Trans. Pattern Anal. Mach. Intell. 32(9), 1627–1645 (2010)CrossRefGoogle Scholar
  9. 9.
    Duda, R.O., Hart, P.E.: Use of the Hough Transformation to Detect Lines and Curves in Pictures. Comm. ACM 15, 11–15 (1972)CrossRefzbMATHGoogle Scholar
  10. 10.
    Deng, Y., Manjunath, B.S.: Unsupervised segmentation of color-texture regions in images and video. IEEE Trans. Pattern Anal. Mach. Intell. 23(8), 800–810Google Scholar
  11. 11.
    Bettadapura, V., Thomaz, E., Parnami, A., Abowd, G.D., Essa, I.A.: Leveraging Context to Support Automated Food Recognition in Restaurants. In: WACV 2015, pp. 580–587Google Scholar
  12. 12.
    Arbelaez, P., Maire, M., Fowlkes, C., Malik, J.: Contour detection and hierarchical image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 33(5), 898–916 (2011)CrossRefGoogle Scholar
  13. 13.
    Puri, M., Zhu, Z., Yu, Q., Divakaran, A., Sawhney, H.: Recognition and volume estimation of food intake using a mobile device. In: IEEE WACV, pp. 1–8 (2009)Google Scholar
  14. 14.
    Cai, W., Yu, Q., Wang, H., Zheng, J.: A fast contour-based approach to circle and ellipse detection. In: 5th IEEE WCICA (2004)Google Scholar
  15. 15.
    Anthimopoulos, M., Dehais, J., Diem, P., Mougiakakou, S.: Segmentation and recognition of multi-food meal images for carbohydrate counting. In: IEEE BIBE (2013)Google Scholar
  16. 16.
    Kawano, Y., Yanai, K.: FoodCam: A real-time food recognition system on a smartphone. Multimedia Tools and Applications, pp. 1–25 (2014)Google Scholar
  17. 17.
    Rother, C., Kolmogorov, V., Blake, A.: GrabCut: Interactive foreground extraction using iterated graph cuts. ACM Trans. Graph. 23, 309–314 (2004)CrossRefGoogle Scholar
  18. 18.
    Morikawa, C., Sugiyama, H., Aizawa, K.: Food region segmentation in meal images using touch points. In: ACM Workshop on Multimedia for Cooking And Eating Activities (2012)Google Scholar
  19. 19.
    Oliveira, L., Costa, V., Neves, G., Oliveira, T., Jorge, E., Lizarraga, M.: A mobile, lightweight, poll-based food identification system. Pattern Recognition 47, 1941–1952 (2014)CrossRefGoogle Scholar
  20. 20.
    Canny, J.: A Computational Approach to Edge Detetion. IEEE Trans. Pattern Analysis and Machine Intelligence 8(6), 679–698 (1986)CrossRefGoogle Scholar
  21. 21.
    Fischler, M.A., Bolles, R.C.: Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography. Comm. of the ACM 24(6), 381–395 (1981)CrossRefMathSciNetGoogle Scholar
  22. 22.
    Ni, K., Jin, H., Dellaert, F.: Groupsac: efficient consensus in the presence of groupings. In: IEEE 12th International Conference on Computer Vision, pp. 2193–2200 (2009)Google Scholar
  23. 23.
    Chum, O., Matas, J., Kittler, J.: Locally optimized RANSAC. In: Michaelis, B., Krell, G. (eds.) DAGM 2003. LNCS, vol. 2781, pp. 236–243. Springer, Heidelberg (2003)CrossRefGoogle Scholar
  24. 24.
    Adams, R., Bischof, L.: Seeded region growing. IEEE Transactions on Pattern Analysis and Machine Intelligence 16(6), 641–647 (1994)CrossRefGoogle Scholar
  25. 25.
    McDonald, R., Smith, K.J.: CIE94a new color difference formula. Journal of the Society of Dyers and Colourists 111(12), 376–379 (1995)CrossRefGoogle Scholar
  26. 26.
    Nock, R., Nielsen, F.: Statistical region merging. IEEE Trans. Pattern Analysis and Machine Intelligence 26(11), 1452–1458 (2004)CrossRefGoogle Scholar
  27. 27.
    Huang, Q., Dom, B.: Quantitative methods of evaluating image segmentation. International Conference on Image Processing 3, 53–56 (1995)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Joachim Dehais
    • 1
    • 2
    Email author
  • Marios Anthimopoulos
    • 1
    • 3
  • Stavroula Mougiakakou
    • 1
    • 4
  1. 1.ARTORG Center for Biomedical Engineering ResearchUniversity of BernBernSwitzerland
  2. 2.Graduate School of Cellular and Biomedical SciencesUniversity of BernBernSwitzerland
  3. 3.Department of Emergency MedicineBern University HospitalBernSwitzerland
  4. 4.Department of Endocrinology, Diabetes and Clinical NutritionBern University HospitalBernSwitzerland

Personalised recommendations