The Use of Temporal Information in Food Image Analysis

  • Yu WangEmail author
  • Ye He
  • Fengqing Zhu
  • Carol Boushey
  • Edward Delp
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9281)


We have developed a dietary assessment system that uses food images captured by a mobile device. Food identification is a crucial component of our system. Achieving a high classification rates is challenging due to the large number of food categories and variability in food appearance. In this paper, we propose to improve food classification by incorporating temporal information. We employ recursive Bayesian estimation to incrementally learn from a person’s eating history. We show an improvement of food classification accuracy by 11% can be achieved.


Mobile Device Temporal Information Accuracy Improvement Food Category Temporal Context 
These keywords were added by machine and not by the authors. This process is experimental and the keywords may be updated as the learning algorithm improves.


  1. 1.
    The TADA project.
  2. 2.
    Ahmad, Z., Khanna, N., Kerr, D., Boushey, C., Delp, E.: A mobile phone user interface for image-based dietary assessment. In: Proceedings of the IS&T/SPIE Conference on Mobile Devices and Multimedia 9030, 903007-1-9, San Francisco, CA (February 2014)Google Scholar
  3. 3.
    Arulampalam, S., Maskell, S., Gordon, N., Clapp, T.: A tutorial on particle filters for online nonlinear/non-gaussian bayesian tracking. IEEE Transactions on Signal Processing 50(2), 174–188 (2002)CrossRefGoogle Scholar
  4. 4.
    Bar, M.: Visual objects in context. Nature Reviews Neuroscience 5(8), 617–629 (2004)CrossRefMathSciNetGoogle Scholar
  5. 5.
    Beijbom, O., Joshi, N., Morris, D., Saponas, S., Khullar, S.: Menu-match: Restaurant-specific food logging from images. In: Winter Conference on Applications of Computer Vision, pp. 844–851, Waikoloa, HI (January 2015)Google Scholar
  6. 6.
    Eicher-Miller, H., Boushey, C.: The most frequently reported foods and beverages differ by age among participants of nhanes 1999–2008. The Journal of the Federation of American Societies for Experimental Biology 26, 256–261 (2012)Google Scholar
  7. 7.
    Fagot-Campagna, A., Saaddine, J., Flegal, K., Beckles, G.: Diabetes, impaired fasting glucose, and elevated hba1c in us adolescents: the third national health and nutrition examination survey. Diabetes Care 24(5), 834–837 (2001)CrossRefGoogle Scholar
  8. 8.
    Felzenszwalb, P., Huttenlocher, D.: Image segmentation using local variation. In: Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition, pp. 98–104, Santa Barbara, CA (June 1998)Google Scholar
  9. 9.
    He, Y., Xu, C., Khanna, N., Boushey, C., Delp, E.: Context based food image analysis. In: Proceedings of IEEE International Conference on Image Processing, pp. 2748–2752, Melbourne, Australia (September 2013)Google Scholar
  10. 10.
    He, Y., Xu, C., Khanna, N., Boushey, C., Delp, E.: Food image analysis: segmentation, identification and weight estimation. In: Proceeding of the IEEE International Conference on Multimedia and Expo, pp. 1–6, San Jose, CA (July 2013)Google Scholar
  11. 11.
    He, Y., Xu, C., Khanna, N., Boushey, C., Delp, E.: Analysis of food images: Features and classification. In: Proceedings of the IEEE International Conference on Image Processing, pp. 2744–2748, Paris, France (October 2014)Google Scholar
  12. 12.
    Manjunath, B., Ohm, J.R., Vasudevan, V., Yamada, A.: Color and texture descriptors. IEEE Transactions on Circuits and Systems for Video Technology 11(6), 703–715 (2001)CrossRefGoogle Scholar
  13. 13.
    Matsuda, Y., Yanai, K.: Multiple-food recognition considering co-occurrence employing manifold ranking. In: Proceedings of the IEEE International Conference on Pattern Recognition, pp. 2017–2020, Tsukuba, Japan (November 2012)Google Scholar
  14. 14.
    McFee, B., Galleguillos, C., Lanckriet, G.: Contextual object localization with multiple kernel nearest-neighbor. IEEE Transactions on Image Processing 20(2), 570–585 (2011)Google Scholar
  15. 15.
    Ogden, C., Carroll, M., Curtin, L., Lamb, M., Flegal, K.: Prevalence of high body mass index in us children and adolescents, 2007–2008. Jama 303(3), 242–249 (2010)CrossRefGoogle Scholar
  16. 16.
    Rabinovich, A., Vedaldi, A., Galleguillos, C., Wiewiora, E., Belongie, S.: Objects in context. In: Proceedings of IEEE International Conference on Computer vision, pp. 1–8, Rio de Janeiro, Brazil (October 2007)Google Scholar
  17. 17.
    Sarkka, S.: Bayesian Filtering and Smoothing. Cambridge University Press, Cambridge (2013)Google Scholar
  18. 18.
    Zhu, F., Bosch, M., Woo, I., Kim, S., Boushey, C., Ebert, D., Delp, E.: The use of mobile devices in aiding dietary assessment and evaluation. IEEE Journal of Selected Topics in Signal Processing 4(4), 756–766 (2010)CrossRefGoogle Scholar
  19. 19.
    Zhu, F., Bosch, M., Khanna, N., Boushey, C., Delp, E.: Multiple hypotheses image segmentation and classification with application to dietary assessment. IEEE Journal of Biomedical and Health Informatics 19(1), 377–388 (2015)CrossRefGoogle Scholar

Copyright information

© Springer International Publishing Switzerland 2015

Authors and Affiliations

  • Yu Wang
    • 1
    Email author
  • Ye He
    • 2
  • Fengqing Zhu
    • 1
  • Carol Boushey
    • 3
  • Edward Delp
    • 1
  1. 1.School of Electrical and Computer EngineeringPurdue UniversityWest LafayetteUSA
  2. 2.GoogleMountain ViewUSA
  3. 3.Cancer Epidemiology ProgramUniversity of Hawaii Cancer CenterHonoluluUSA

Personalised recommendations