Design in Everyday Cooking: Challenges for Assisting with Menu Planning and Food Preparation

  • Atsushi HashimotoEmail author
  • Jun Harashima
  • Yoko Yamakata
  • Shinsuke Mori
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 9741)


In this study, we introduce challenges for assisting with everyday cooking activities. Menu planning is the first step in daily cooking, and there are many commercial services available. We introduce the case study of “cookpad,” one of the largest recipe portal sites, and illustrate their efforts to maintain an up-to-date recipe search system. As an academic challenge, situated recipe recommendation is also introduced. Food preparation is another important topic. We present our perspective based on the relationship between recipe texts and cooking activities, along with related studies.


Recipe Cooking activity 

1 Introduction

Cooking is a fundamental activity in our daily lives. A good meal enriches our quality of life, and it promotes wellness as well as provides pleasure. It can act to ease family budgets. In some cases, a meal can have specific religious or cultural meanings. To address these multifaceted needs, various types of improvement to cooking and meal planning might be possible by designing systems with the support of information and communication technology.

Daily cooking activities include regular repetition of planning a menu, preparing food, and eating. For eating, the main target for improvement concerns health administration [1, 2, 3, 4, 5, 6]. The challenge of these studies is to recognize a menu and estimate a user’s nutritional intake from a photo taken by the user’s mobile phone. To encourage good eating habits, Takeuchi et al. [7] designed an interactive tool on a social network service; users can share photos of their meals and receive remarks from their friends via this tool. This communication tool is designed to prompt users to make healthier menu choices, by secretly sorting positive remarks based on the healthiness of the meal.

Although there are a multitude of interesting studies concerning eating, it is difficult to introduce them all. In this study, we focus on recent attempts to design modern applications for assisting with menu planning and food preparation, as these are activities of interest for us.

There are many recipe portal sites, and they are accessible from all over the world, because menu planning is a task that is directly related to people’s purchasing behavior. In spite of the universal need of recipe sites, there are large cultural differences in recipes. As is clear from the fact that “Washoku” (Japanese cuisine) won world heritage status in 2013, cooking has deep cultural aspects. Religions, histories, climates, and industrial progress are all related to local foods. A typical cross-cultural problem is translation of recipes, which is one of the desired applications. Ingredients, culinary arts, cooking devices, and equipment are subject to translation as well as language. In this sense, cooking is a challenging target for cross-cultural computing.

This manuscript is organized as follows. Section 2 introduces the technical efforts of a recipe portal site “cookpad”1 as a case study of a commercial assistive service for menu planning. Academic challenges for menu planning are overviewed in Sect. 3. Section 4 summarizes assistive systems for food preparation from the viewpoint of the relationship between recipe texts and human actions. Finally, Sect. 5 presents the conclusion.

2 Challenges in a Company

In this section, we describe challenges faced by cookpad, an Internet site comprising over 2.2 million recipes as of January 2016, making it one of the largest recipe sites in the world. Of the many challenges faced by the site, we specifically introduce those that are closely related to academic research.

Recipe Search. Like general recipe sites, cookpad provides users with a search box to help them find recipes efficiently. Because recipes in cookpad are written in Japanese, which is a language that does not delimit words by white-spaces, morphological analysis is necessary to recognize words in the recipes. The analyzer in cookpad uses a manually maintained dictionary consisting of a vast number of food-related words. In the recipe search, synonymous expressions are also recognized using a domain-specific synonym dictionary, which is also maintained manually.

Recipe Classification. To find recipes, it is not sufficient merely to provide a search box. In cookpad, recipes are automatically classified into various categories (e.g., meat dishes, seafood dishes, vegetable dishes) and users can limit their search results using this information. The classification is based on a machine learning method (using support vector machines) where tens of thousands of recipes are used as labeled data. To ensure service quality, the precision for each category is maintained at \(90\,\%\) or above so that users can find relevant recipes easily.

Content Selection. Search results consisting only of recipes may not always satisfy the user’s information need. According to users’ queries, major search engines show not only web pages but also other content (e.g., YouTube videos in Google search results). Likewise, the cookpad shows not only recipes but also various other content (e.g., tips, news, and videos related to the user’s queries). For each query, content is selected based on a multi-armed bandit algorithm to maximize its click-through rate.

Research Promotion. To promote food research, cookpad has made its recipes available [8] through the National Institute of Informatics (NII), a Japanese Research Institute with the goal of advancing informatics research. The 1.7 million recipes involved in this challenge are those that were uploaded to cookpad by the end of September 2014. This data collection was released in February 2015.2 Any researcher in public institutions (e.g., university) can obtain access to the collection for research purposes and as of January 2016, 82 research groups at 56 universities had already done so.

3 Academic Challenges for a Smart Recipe Search

3.1 Cooking Recipe Recommendations Using Surrounding Information

One of the most stressful issues for homemakers is to decide the day’s menu [9]. Many homemakers browse the web to decide what they are going to cook each day. Although most existing recipe search systems request that the user submits a query, such as the name of the menu that they want or names of the ingredients that they want to use [10, 11, 12], people cannot always explain the property of the recipe that they are looking for, even if they have a preference for a recipe. To deal with these issues, we considered that what people want to eat must strongly depend on their daily circumstances and that a recipe cooked in a given situation must be preferred by someone who is in a similar situation. Therefore, we have proposed a method that recommends recipes not only according to a recipe’s properties but also according to the user’s situation.

On the web, there are many blog-type recipes that describe not only the recipe itself but also the reason why the recipe was selected. We analyzed 2,074 blog-type recipes which were randomly collected from “RECIPE BLOG” [13]. We found that 48.2 % blog-type recipes describe at least one reason for selecting recipes and that these reasons could be classified into 18 categories. We then devised an algorithm that extracts situations corresponding to the 18 categories from a user’s life log, such as his or her tweets, and recommends recipes that have similar reasons to the extracted ones. We evaluated the proposed method under three fictitious scenarios; in two of them, the user had vague requirements for recipes and they could only describe their situation as “it was so busy today.” We call it as “situational scenario.” The other had concrete requirements such as “I bought Pacific saury for a good price.” We call this a “procedural scenario.” We input the description of each scenario into the proposed system and obtained the top ten results. As a baseline method, we also made queries derived from each scenario and found ten recipes that contained the queries as a part of the recipes. Five examinees evaluated each recipe recommended by the two methods using a five-point scale. The proposed method obtained a higher score than the baseline method in the two situational scenarios, while the baseline method obtained a higher score in the procedural one. Therefore, the proposed method is useful when a user cannot make his or her requirements clear as mentioned above. For details, please refer to [14].

3.2 Recipe Comparison Using a Recipe Flow-Graph

As will be shown in the next section, we are constructing a method to translate the procedural text of a recipe into a flow-graph, which will represent the procedural workflow of a recipe, such as which cooking actions are performed to which ingredients, and which ingredients are mixed with each other. Although the flow-graph introduced in the next section can be described as a Directed Acyclic Graph (DAG), it can be simplified to an ordered labeled tree, as shown in Fig. 1, which we call a recipe tree, as we focus on the flow related to ingredients, tools and cooking actions. A recipe tree has significant advantages over a recipe flow-graph because it has multiple fast and effective graph-matching algorithms, including the editing distance calculation. Therefore, we have proposed a method to calculate similarities (distance) between two recipes by calculating the editing distance between their recipe trees.
Fig. 1.

A recipe tree of one “NIKUJAGA” recipe.

In general for unordered labeled trees, including recipe trees, the problem of computing the editing distance between two trees is difficult (more precisely, the problem is known to be NP-hard [15]). Therefore, we developed heuristic algorithms specialized for cooking recipe trees. The proposed method not only calculates the distance between two recipes but also derives the details of their differences. Figure 2 shows the mapping result between two “NIKUJAGA” recipes when one recipe is mapped onto the other, with minimal operation costs. According to the result, we can easily note differences such as “the timing of mixing meat differs between the two recipes” and the action “immerse the cut potato” in the left recipe is omitted in the right one. Moreover, we have proposed a method to find a common cooking flow of a specific dish by extracting the common structures of recipe trees that are translated from the top ten search results from the name of that specific dish. The details of these methods are provided in [16]. In a related study, Wang et al. [17] also proposed flow-graph representations of recipes. They defined the similarity between two recipes based on the number of common subgraphs shared by two recipes.
Fig. 2.

Result of mapping one “Nikujaga” recipe to another.

4 Assistive Applications for Food Preparation

Recipes are important information sources in food preparation. People prefer to eat a different menu everyday. Some external factors, such as childbirth or a sick family member, may force us to change our repertoire of dishes. Both a variety of recipes and comprehensive recipe presentation are important to enrich our food preparation experience.
Fig. 3.

Translation of Recipe/Observation into an intermediate model and its applications.

Figure 3 shows our image of the applications and agendas in this topic. This perspective is inspired by machine translation via an intermediate model; this is, once we map the recipe text and observed food preparation activities onto a semantic model, we can directly compare the text and activities. In this sense, the semantic model of the working process corresponds to an intermediate language model in the machine translation problem.

A typical application is recipe generation from observation, which is achieved by agendas C\(\rightarrow \)B in Fig. 3. There are many challenges to recognizing food preparation activities (agenda C) [18, 19, 20, 21, 22, 23, 24, 25, 26, 27] and some to generating recipe texts (agenda B) [28].

Recipe translation is realized by A\(\rightarrow \)D\(\rightarrow \)B. Agenda D has not yet been studied in detail; a comparison in the structured recipe representation in Fig. 2 is the first step of D, i.e. this study is front-line research toward this goal. As mentioned in Sect. 3.2, the DAG (or tree) representation of a recipe is a natural structural organization of the cooking process, and some research teams have proposed methods to extract the structure from recipe texts (agenda A) [29, 30, 31]. Most of the challenges facing processes A and B are addressed in the Japanese language [16, 30, 31], and but little in other languages [29]. For recipe translation and localization, it is important to develop these techniques in multiple languages.

An example of a workflow structure provided by [30] is shown in Fig. 4 with vertex labels (Table 1) and edge labels (Table 2). This is the flow extracted from a recipe posted on cookpad. Because recipes on cookpad are user-generated, texts are informal and there are many difficulties: disparate notation, non-regularized manners of reference, and broken contexts of descriptions.
Fig. 4.

An example of flowgraph.

To extract the complete structure from informal recipes, it might be useful to complement the recipes with information derived from observation. One of our recent trials attempted to match recipe texts and cooking observations in the semantic model to obtain reliable and informative structural representations of the recipes. The number of matched pairs results in statistical information concerning human activities, i.e., the human tendency to cook a dish following a recipe. Such information should enhance performance in systems involving agendas A-E.
Table 1.

Named entity tags.

NE tag











Action by the chef


Action by foods


State of foods


State of tools

Table 2.

Edge labels.

Edge label



Action agent


Action target


Action destination


Food complement


Tool complement


Food equality


Food part-of


Food set


Tool equality


Tool part-of


Action equality


Head verb of a clause for timing, etc.


Other relationships

While the above applications contribute to increasing the variety of recipes, there are a series of proposals to improve recipe presentation [18, 32, 33, 34, 35, 36, 37, 38]. Most of them [32, 33, 34, 35, 37, 38] assume a simple recipe structure, which is a sequence of steps. A few other studies use the DAG or tree structure for a more intelligent recipe presentation [18, 36].

In [18], we have forecasted the next intended step of a user via human-object interaction on a cooking surface. A human interacts with objects to proceed with cooking tasks. Simultaneously, the human puts objects aside if they are in his or her way. The challenge was to identify the intended step even while such out-of-context interactions are observed with informative interactions. The proposed method estimates the progress of the cooking process from the history of the interactions together with the intended next step. The estimated progress narrows down the options for the next step. In the experiment, the proposed method achieved more than 70 % accuracy in its forecast. Because forecasting is generally a difficult task of pattern recognition, this is a remarkable score, nonetheless the accuracy should be enhanced by more sophisticated semantic models with statistical information.

5 Conclusions

In this study, we introduced challenges for assisting with everyday cooking activities, particularly menu planning and food preparation. There are many commercial websites that provide recipes. Accurate and easy-to-use recipe search and recommendation tools are important to increase the number of customers. We introduced challenges by cookpad as a practical example. An accurate and large corpus is critical to achieve a practical recipe search engine. Companies pay a lot of efforts to maintain their corpus and keep the system up-to-date.

We also introduced academic challenges for menu planning. There is a lot of room for academic researchers to enhance recipe recommendation. Food preferences are influenced by the user’s day; therefore, a highly customized recipe recommendation requires a daily context of the family members. From this perspective, both web mining and Life logging are within the scope of this application.

While there have been several assistive systems and related methods for food preparation, one of the most important challenges is to obtain a highly informative semantic model of the food preparation process. We believe that matching the recipe text and cooking observations will provide a breakthrough in this topic. Finally, more challenges in non-Japanese languages are waited for cross-cultural applications in everyday cooking.




This work was supported by JSPS KAKENHI Grant Numbers 24240030, 26280039, 26280084.


  1. 1.
    Wang, X., Kumar, D., Thome, N., Cord, M., Precioso, F.: Recipe recognition with large multimodal food dataset. In: Proceedings of IEEE International Conference on Multimedia & Expo Workshops, pp. 1–6 (2015)Google Scholar
  2. 2.
    Anthimopoulos, M., Gianola, L., Scarnato, L., Diem, P., Mougiakakou, S.: A food recognition system for diabetic patients based on an optimized bag of features approach. IEEE J. Biomed. Health Inform. 18(4), 1261–1271 (2014)CrossRefGoogle Scholar
  3. 3.
    Sudo, K., Murasaki, K., Shimamura, J., Taniguchi, Y.: Estimating nutritional value from food images based on semantic segmentation. In: Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication. UbiComp 2014 Adjunct, pp. 571–576 (2014)Google Scholar
  4. 4.
    Kitamura, K., Yamasaki, T., Aizawa, K.: Foodlog: capture, analysis and retrieval of personal food images via web. In: Proceedings of the ACM Multimedia 2009 Workshop on Multimedia for Cooking and Eating Activities, pp. 23–30 (2009)Google Scholar
  5. 5.
    Khanna, N., Boushey, C.J., Kerr, D., Okos, M., Ebert, D.S., Delp, E.J.: An overview of the technology assisted dietary assessment project at purdue university. In: Proceedings of 2010 IEEE International Symposium on Multimedia, pp. 290–295 (2010)Google Scholar
  6. 6.
    Kawano, Y., Yanai, K.: Food image recognition with deep convolutional features. In: Proceedings of ACM UbiComp Workshop on Workshop on Smart Technology for Cooking and Eating Activities (CEA), September 2014Google Scholar
  7. 7.
    Takeuchi, T., Fujii, T., Narumi, T., Tanikawa, T., Hirose, M.: Considering individual taste in social feedback to improve eating habits. In: Proceedings of IEEE International Conference on Multimedia & Expo Workshops, pp. 1–6 (2015)Google Scholar
  8. 8.
    Harashima, J., Ariga, M., Murata, K., Ioki, M.: A large-scale recipe and meal data collection as infrastructure for food research. In: Proceedings of the 10th International Conference on Language Resources and Evaluation (2016, to appear)Google Scholar
  9. 9.
    Mynavi Corporation: Cooking related questionary investi-gation reported by Mynavi woman on 27th (in Japanese). Accessed 1 Feb 2016
  10. 10.
    Oyama, S., Kokubo, T., Ishida, T.: Domain-specific web search with keyword spices. IEEE Trans. Knowl. Data Eng. 16(1), 17–27 (2004)CrossRefGoogle Scholar
  11. 11.
    Tsukuda, K., Yamamoto, T., Nakamura, S., Tanaka, K.: Plus one or minus one: a method to browse from an object to another object by adding or deleting an element. In: Bringas, P.G., Hameurlain, A., Quirchmayr, G. (eds.) DEXA 2010, Part II. LNCS, vol. 6262, pp. 258–266. Springer, Heidelberg (2010)CrossRefGoogle Scholar
  12. 12.
    Chung, Y.: Finding food entity relationships using user-generated data in recipe service. In: Proceedings of the 21st ACM International Conference on Information and Knowledge Management, pp. 2611–2614 (2012)Google Scholar
  13. 13.
    Ai-Land Co., Ltd.: Recipe blog (in Japanese). Accessed 1 Feb 2016
  14. 14.
    Kadowaki, T., Mori, S., Yamakata, Y., Tanaka, K.: Recipe search for blog-type recipe articles based on a users situation. In: Proceedings of ACM Conference on Ubiquitous Computing, pp. 497–506 (2014)Google Scholar
  15. 15.
    Zhang, K., Jiang, T.: Some MAX SNP-hard results concerning unordered labeled trees. Inf. Process. Lett. 49(5), 249–254 (1994)MathSciNetCrossRefzbMATHGoogle Scholar
  16. 16.
    Yamakata, Y., Imahori, S., Sugiyama, Y., Mori, S., Tanaka, K.: Feature extraction and summarization of recipes using flow graph. In: Jatowt, A., et al. (eds.) SocInfo 2013. LNCS, vol. 8238, pp. 241–254. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  17. 17.
    Wang, L., Li, Q., Li, N., Dong, G., Yang, Y.: Substructure similarity measurement in chinese recipes. In: Proceedings of the 17th International Conference on World Wide Web, pp. 979–988 (2008)Google Scholar
  18. 18.
    Hashimoto, A., Inoue, J., Funatomi, T., Minoh, M.: How does user’s access to object make HCI smooth in recipe guidance? In: Rau, P.L.P. (ed.) CCD 2014. LNCS, vol. 8528, pp. 150–161. Springer, Heidelberg (2014)Google Scholar
  19. 19.
    Shimada, A., Kondo, K., Deguchi, D., Morin, G., Stern, H.: Kitchen scene context based gesture recognition: a contest in ICPR2012. In: Jiang, X., Bellon, O.R.P., Goldgof, D., Oishi, T. (eds.) WDIA 2012. LNCS, vol. 7854, pp. 168–185. Springer, Heidelberg (2013)CrossRefGoogle Scholar
  20. 20.
    Iscen, A., Duygulu, P.: Knives are picked before slices are cut: recognition through activity sequence analysis. In: Proceedings of the 5th International Workshop on Multimedia for Cooking and Eating Activities, pp. 3–8 (2013)Google Scholar
  21. 21.
    Rohrbach, M., Amin, S., Andriluka, M., Schiele, B.: A database for fine grained activity detection of cooking activities. In: Proceedings of 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1194–1201 (2012)Google Scholar
  22. 22.
    Packer, B., Saenko, K., Koller, D.: A combined pose, object, and feature model for action understanding. In: Proceedings of 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1378–1385 (2012)Google Scholar
  23. 23.
    Lei, J., Ren, X., Fox, D.: Fine-grained kitchen activity recognition using RGB-D. In: Proceedings of the 2012 ACM Conference on Ubiquitous Computing, pp. 208–211 (2012)Google Scholar
  24. 24.
    Hashimoto, A., Inoue, J., Nakamura, K., Funatomi, T., Ueda, M., Yamakata, Y., Minoh, M.: Recognizing ingredients at cutting process by integrating multimodal features. In: Proceedings of the ACM Multimedia 2012 Workshop on Multimedia for Cooking and Eating Activities, pp. 13–18 (2012)Google Scholar
  25. 25.
    Ueda, M., Funatomi, T., Hashimoto, A., Watanabe, T., Minoh, M.: Developing a real-time system for measuring the consumption of seasoning. In: Proceedings of IEEE ISM 2011 Workshop on Multimedia for Cooking and Eating Activities, pp. 393–398 (2011)Google Scholar
  26. 26.
    Hashimoto, A., Mori, N., Funatomi, T., Mukunoki, M., Kakusho, K., Minoh, M.: Tracking food materials with changing their appearance in food preparing. In: Proceedings of ISM 2010 Workshop on Multimedia for Cooking and Eating Activities, pp. 248–253. IEEE (2010)Google Scholar
  27. 27.
    Miyawaki, K., Sano, M.: A virtual agent for a cooking navigation system using augmented reality. In: Prendinger, H., Lester, J.C., Ishizuka, M. (eds.) IVA 2008. LNCS (LNAI), vol. 5208, pp. 97–103. Springer, Heidelberg (2008)CrossRefGoogle Scholar
  28. 28.
    Yamasaki, T., Yoshino, K., Maeta, H., Sasada, T., Hashimoto, A., Funatomi, T., Yamakata, Y., Mori, S.: Procedual text generation from a flow graph. IPSJ J. 57(3) (to appear). Written in JapaneseGoogle Scholar
  29. 29.
    Kiddon, C., Ponnuraj, G.T., Zettlemoyer, L., Choi, Y.: Mise en place: unsupervised interpretation of instructional recipes. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 982–992 (2015)Google Scholar
  30. 30.
    Maeta, H., Sasada, T., Mori, S.: A framework for procedural text understanding. In: Proceedings of the 14th International Conference on Parsing Technologies (2015)Google Scholar
  31. 31.
    Karikome, S., Fujii, A.: Improving structural analysis of cooking recipe text. IEICE Tech. Rep. Data Eng. 112(75), 43–48 (2012)Google Scholar
  32. 32.
    Sato, A., Watanabe, K., Rekimoto, J.: Shadow cooking: situated guidance for a fluid cooking experience. In: Stephanidis, C., Antona, M. (eds.) UAHCI 2014, Part III. LNCS, vol. 8515, pp. 558–566. Springer, Heidelberg (2014)Google Scholar
  33. 33.
    Matsushima, Y., Funabiki, N., Zhang, Y., Nakanishi, T., Watanabe, K.: Extensions of cooking guidance function on android tablet for homemade cooking assistance system. In: IEEE 2nd Global Conference on Consumer Electronics, pp. 397–401 (2013)Google Scholar
  34. 34.
    Halupka, V., Almahr, A., Pan, Y., Cheok, A.D.: Chop chop: a sound augmented kitchen prototype. In: Nijholt, A., Romão, T., Reidsma, D. (eds.) ACE 2012. LNCS, vol. 7624, pp. 494–497. Springer, Heidelberg (2012)CrossRefGoogle Scholar
  35. 35.
    Uriu, D., Namai, M., Tokuhisa, S., Kashiwagi, R., Inami, M., Okude, N.: Panavi: recipe medium with a sensors-embedded pan for domestic users to master professional culinary arts. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 129–138 (2012)Google Scholar
  36. 36.
    Hamada, R., Okabe, J., Ide, I., Sakai, S., Tanaka, H.: Cooking navi: assistant for daily cooking in kitchen. In: Proceedings of the 13th Annual ACM International Conference on Multimedia, pp. 371–374 (2005). Written in JapaneseGoogle Scholar
  37. 37.
    Bradbury, J.S., Shell, J.S., Knowles, C.B.: Hands on cooking: towards an attentive kitchen. In: Proceedings of CHI 2003 Extended Abstracts on Human Factors in Computing Systems, pp. 996–997 (2003)Google Scholar
  38. 38.
    Ju, W., Hurwitz, R., Judd, T., Lee, B.: Counteractive: an interactive cookbook for the kitchen counter. In: Proceedings of CHI 2001 Extended Abstracts on Human Factors in Computing Systems, pp. 269–270. ACM, New York (2001)Google Scholar

Copyright information

© Springer International Publishing Switzerland 2016

Authors and Affiliations

  • Atsushi Hashimoto
    • 1
    Email author
  • Jun Harashima
    • 2
  • Yoko Yamakata
    • 3
  • Shinsuke Mori
    • 1
  1. 1.Kyoto UniversitySakyo-kuJapan
  2. 2.Cookpad IncShibuya-kuJapan
  3. 3.The University of TokyoBunkyo-kuJapan

Personalised recommendations