Towards Natural Language Understanding of Procedural Text Using Recipes

  • Dena F. Mujtaba
  • Nihar R. MahapatraEmail author
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 1119)


Procedural knowledge, or how-to knowledge, is the knowledge acquired from natural language understanding of instructions in procedural text. Procedural knowledge bases containing textual descriptions of tasks in procedures have witnessed explosive growth recently. This has facilitated a significant body of work in various natural language understanding tasks. A rich source of procedural text is in the form of recipes describing food preparation procedures. The ready availability of online recipes has enabled progress in food computing, which refers to computing tasks related to recipes, such as food perception, recipe image recognition and calorie estimation, and food-oriented retrieval of recipes. However, past work on food computing has not covered the procedural knowledge inherent in recipes and the natural language understanding tasks required to uncover that knowledge. We seek to address this by presenting an overview of recent work in natural language understanding tasks in food computing and describing how this contributes to how-to knowledge and future applications.


Artificial intelligence Natural language processing Natural language understanding Procedural knowledge Food computing Information extraction Recipe representation 



This material is based upon work partly supported by the U.S. National Science Foundation under Grant No. 1936857.


  1. 1.
    Chu, C.X., Tandon, N., Weikum, G.: Distilling task knowledge from how-to communities. In: Proceedings of the 26th International Conference on World Wide Web, pp. 805–814. International World Wide Web Conferences Steering Committee (2017)Google Scholar
  2. 2.
    Hune-Brown, N.: Allrecipes reveals the enormous gap between foodie culture and what Americans actually cook (2016).
  3. 3.
    Harper, C., Siller, M.: OpenAG: a globally distributed network of food computing. IEEE Pervasive Comput. 14(4), 24–27 (2015)CrossRefGoogle Scholar
  4. 4.
    Min, W., Jiang, S., Liu, L., Rui, Y., Jain, R.: A survey on food computing. ACM Comput. Surv. (CSUR) 52(5), 92 (2019)CrossRefGoogle Scholar
  5. 5.
    Min, W., Jiang, S., Jain, R.: Food recommendation: Framework, existing solutions and challenges (2019). ArXiv:1905.06269
  6. 6.
    Ofli, F., Aytar, Y., Weber, I., Al Hammouri, R., Torralba, A.: Is saki #delicious? the food perception gap on Instagram and its relation to health. In: Proceedings of the 26th International Conference on World Wide Web, pp. 509–518. International World Wide Web Conferences Steering Committee (2017)Google Scholar
  7. 7.
    Wang, X., Kumar, D., Thome, N., Cord, M., Precioso, F.: Recipe recognition with large multimodal food dataset. In: 2015 IEEE International Conference on Multimedia and Expo Workshops (ICMEW), pp. 1–6. IEEE (2015)Google Scholar
  8. 8.
    Theodoridis, T., Solachidis, V., Dimitropoulos, K., Gymnopoulos, L., Daras, P.: A survey on AI nutrition recommender systems. PETRA 540–546 (2019)Google Scholar
  9. 9.
    Marin, J., Biswas, A., Ofli, F., Hynes, N., Salvador, A., Aytar, Y., Weber, I., Torralba, A.: Recipe1M: A dataset for learning cross-modal embeddings for cooking recipes and food images (2018). ArXiv: 1810.06553
  10. 10.
    Minsky, M.: A framework for representing knowledge. MIT-AI laboratory memo 306. Massachusetts Institute of Technology (1974)Google Scholar
  11. 11.
    Baker, C.F., Fillmore, C.J., Lowe, J.B.: The Berkeley frameNet project. In: Proceedings of the 17th International Conference on Computational Linguistics, vol. 1, pp. 86–90. Association for Computational Linguistics (1998)Google Scholar
  12. 12.
    Schuler, K.K.: VerbNet: A broad-coverage, comprehensive verb lexicon (2005)Google Scholar
  13. 13.
    Yordanova, K.Y.: TextToHBM: A generalised approach to learning models of human behaviour for activity recognition from textual instructions. In: Workshops at the Thirty-First AAAI Conference on Artificial Intelligence (2017)Google Scholar
  14. 14.
    Wanzare, L.D., Zarcone, A., Thater, S., Pinkal, M.: A crowdsourced database of event sequence descriptions for the acquisition of high-quality script knowledge. In: Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), pp. 3494–3501 (2016)Google Scholar
  15. 15.
    Zhang, Z., Webster, P., Uren, V.S., Varga, A., Ciravegna, F.: Automatically extracting procedural knowledge from instructional texts using natural language processing. LREC 2012, 520–527 (2012)Google Scholar
  16. 16.
    Schumacher, P., Minor, M., Schulte-Zurhausen, E.: Extracting and enriching workflows from text. In: 2013 IEEE 14th International Conference on Information Reuse and Integration (IRI), pp. 285–292. IEEE (2013)Google Scholar
  17. 17.
    Bollini, M., Tellex, S., Thompson, T., Roy, N., Rus, D.: Interpreting and executing recipes with a cooking robot. Exp. Robot. 481–495. Springer (2013)Google Scholar
  18. 18.
    Maeta, H., Sasada, T., Mori, S.: A framework for procedural text understanding (2015)Google Scholar
  19. 19.
    Hynes, N.: Representation learning of recipes. Ph.D. thesis, Massachusetts Institute of Technology (2017)Google Scholar
  20. 20.
    Korpusik, M., Huang, C., Price, M., Glass, J.: Distributional semantics for understanding spoken meal descriptions. In: 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6070–6074. IEEE (2016)Google Scholar
  21. 21.
    Yagcioglu, S., Erdem, A., Erdem, E., Ikizler-Cinbis, N.: RecipeQA: A challenge dataset for multimodal comprehension of cooking recipes (2018). ArXiv: 1809.00812
  22. 22.
    Yamakata, Y., Tajima, K., Mori, S.: A case study on start-up of dataset construction: In case of recipe named entity corpus. In: 2018 IEEE International Conference on Big Data (Big Data), pp. 3564–3567. IEEE (2018)Google Scholar
  23. 23.
  24. 24.
  25. 25.
    Ninomiya, A., Ozaki, T.: Learning distributed representation of recipe flow graphs via frequent subgraphs. In: Proceedings of the 11th Workshop on Multimedia for Cooking and Eating Activities, pp. 25–28. ACM (2019)Google Scholar
  26. 26.
    Mori, S., Maeta, H., Yamakata, Y., Sasada, T.: Flow graph corpus from recipe texts. LREC 2370–2377 (2014)Google Scholar
  27. 27.
    Chang, M., Guillain, L.V., Jung, H., Hare, V.M., Kim, J., Agrawala, M.: RecipeScape: An interactive tool for analyzing cooking instructions at scale. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, p. 451. ACM (2018)Google Scholar
  28. 28.
    Chen, Y.: A statistical machine learning approach to generating graph structures from food recipes. Ph.D. thesis, Brandeis University (2017)Google Scholar
  29. 29.
    Xie, H., Yu, L., Li, Q.: A hybrid semantic item model for recipe search by example. In: 2010 IEEE International Symposium on Multimedia, pp. 254–259. IEEE (2010)Google Scholar
  30. 30.
    Chen, J.J., Ngo, C.W., Feng, F.L., Chua, T.S.: Deep understanding of cooking procedure for cross-modal recipe retrieval. In: 2018 ACM Multimedia Conference on Multimedia Conference, pp. 1020–1028. ACM (2018)Google Scholar
  31. 31.
    Tasse, D., Smith, N.A.: SOUR CREAM: Toward semantic processing of recipes. Technical Report CMU-LTI-08-005, Carnegie Mellon University, Pittsburgh (2008)Google Scholar
  32. 32.
    Jermsurawong, J., Habash, N.: Predicting the structure of cooking recipes. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 781–786 (2015)Google Scholar
  33. 33.
    TAAABLE: A case-based reasoning system which adapts cooking recipes. Revue des Sciences et Technologies de l’Information-Série RIA: Revue d’Intelligence Artificielle 31(1–2), 207–235 (2017)Google Scholar
  34. 34.
    Freyne, J., Berkovsky, S.: Intelligent food planning: Personalized recipe recommendation. In: Proceedings of the 15th International Conference on Intelligent User Interfaces, pp. 321–324. ACM (2010)Google Scholar
  35. 35.
    Hong, J., Lee, H.: Culinary recipe recommendation based on text analytics. Int. J. Eng. Technol. (UAE) 7(4), 5–6 (2018)CrossRefGoogle Scholar
  36. 36.
    Vivek, M., Manju, N., Vijay, M.: Machine learning based food recipe recommendation system. In: Proceedings of International Conference on Cognition and Recognition, pp. 11–19. Springer (2018)Google Scholar
  37. 37.
    Wang, L., Li, Q., Li, N., Dong, G., Yang, Y.: Substructure similarity measurement in Chinese recipes. In: Proceedings of the 17th International Conference on World Wide Web, pp. 979–988. ACM (2008)Google Scholar
  38. 38.
    Teng, C.Y., Lin, Y.R., Adamic, L.A.: Recipe recommendation using ingredient networks. In: Proceedings of the 4th Annual ACM Web Science Conference, pp. 298–307. ACM (2012)Google Scholar
  39. 39.
    Trattner, C., Elsweiler, D.: Food recommender systems: Important contributions, challenges and future research directions (2017). ArXiv:1711.02760
  40. 40.
    Forbes, P., Zhu, M.: Content-boosted matrix factorization for recommender systems: Experiments with recipe recommendation. RecSys 11, 23–27 (2011)Google Scholar
  41. 41.
    Salvador, A., Drozdzal, M., Giro-i Nieto, X., Romero, A.: Inverse cooking: recipe generation from food images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 10,453–10,462 (2019)Google Scholar
  42. 42.
    Lo, Y.W., Zhao, Q., Ting, Y.H., Chen, R.C.: Automatic generation and recommendation of recipes based on outlier analysis. In: 2015 IEEE 7th International Conference on Awareness Science and Technology (iCAST), pp. 216–221. IEEE (2015)Google Scholar
  43. 43.
    Vairale, V.S., Shukla, S.: Recommendation framework for diet and exercise based on clinical data: A systematic review. In: Data Science and Big Data Analytics, pp. 333–346. Springer (2019)Google Scholar
  44. 44.
    Salvador, A., Hynes, N., Aytar, Y., Marin, J., Ofli, F., Weber, I., Torralba, A.: Learning cross-modal embeddings for cooking recipes and food images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3020–3028 (2017)Google Scholar
  45. 45.
    Liu, X., He, P., Chen, W., Gao, J.: Multi-task deep neural networks for natural language understanding. ArXiv:1901.11504 (2019)

Copyright information

© Springer Nature Singapore Pte Ltd. 2020

Authors and Affiliations

  1. 1.Department of Electrical and Computer Engineering, Michigan State UniversityEast LansingUSA

Personalised recommendations