Design in Everyday Cooking: Challenges for Assisting with Menu Planning and Food Preparation
- 2.5k Downloads
In this study, we introduce challenges for assisting with everyday cooking activities. Menu planning is the first step in daily cooking, and there are many commercial services available. We introduce the case study of “cookpad,” one of the largest recipe portal sites, and illustrate their efforts to maintain an up-to-date recipe search system. As an academic challenge, situated recipe recommendation is also introduced. Food preparation is another important topic. We present our perspective based on the relationship between recipe texts and cooking activities, along with related studies.
KeywordsRecipe Cooking activity
Cooking is a fundamental activity in our daily lives. A good meal enriches our quality of life, and it promotes wellness as well as provides pleasure. It can act to ease family budgets. In some cases, a meal can have specific religious or cultural meanings. To address these multifaceted needs, various types of improvement to cooking and meal planning might be possible by designing systems with the support of information and communication technology.
Daily cooking activities include regular repetition of planning a menu, preparing food, and eating. For eating, the main target for improvement concerns health administration [1, 2, 3, 4, 5, 6]. The challenge of these studies is to recognize a menu and estimate a user’s nutritional intake from a photo taken by the user’s mobile phone. To encourage good eating habits, Takeuchi et al.  designed an interactive tool on a social network service; users can share photos of their meals and receive remarks from their friends via this tool. This communication tool is designed to prompt users to make healthier menu choices, by secretly sorting positive remarks based on the healthiness of the meal.
Although there are a multitude of interesting studies concerning eating, it is difficult to introduce them all. In this study, we focus on recent attempts to design modern applications for assisting with menu planning and food preparation, as these are activities of interest for us.
There are many recipe portal sites, and they are accessible from all over the world, because menu planning is a task that is directly related to people’s purchasing behavior. In spite of the universal need of recipe sites, there are large cultural differences in recipes. As is clear from the fact that “Washoku” (Japanese cuisine) won world heritage status in 2013, cooking has deep cultural aspects. Religions, histories, climates, and industrial progress are all related to local foods. A typical cross-cultural problem is translation of recipes, which is one of the desired applications. Ingredients, culinary arts, cooking devices, and equipment are subject to translation as well as language. In this sense, cooking is a challenging target for cross-cultural computing.
This manuscript is organized as follows. Section 2 introduces the technical efforts of a recipe portal site “cookpad”1 as a case study of a commercial assistive service for menu planning. Academic challenges for menu planning are overviewed in Sect. 3. Section 4 summarizes assistive systems for food preparation from the viewpoint of the relationship between recipe texts and human actions. Finally, Sect. 5 presents the conclusion.
2 Challenges in a Company
In this section, we describe challenges faced by cookpad, an Internet site comprising over 2.2 million recipes as of January 2016, making it one of the largest recipe sites in the world. Of the many challenges faced by the site, we specifically introduce those that are closely related to academic research.
Recipe Search. Like general recipe sites, cookpad provides users with a search box to help them find recipes efficiently. Because recipes in cookpad are written in Japanese, which is a language that does not delimit words by white-spaces, morphological analysis is necessary to recognize words in the recipes. The analyzer in cookpad uses a manually maintained dictionary consisting of a vast number of food-related words. In the recipe search, synonymous expressions are also recognized using a domain-specific synonym dictionary, which is also maintained manually.
Recipe Classification. To find recipes, it is not sufficient merely to provide a search box. In cookpad, recipes are automatically classified into various categories (e.g., meat dishes, seafood dishes, vegetable dishes) and users can limit their search results using this information. The classification is based on a machine learning method (using support vector machines) where tens of thousands of recipes are used as labeled data. To ensure service quality, the precision for each category is maintained at \(90\,\%\) or above so that users can find relevant recipes easily.
Content Selection. Search results consisting only of recipes may not always satisfy the user’s information need. According to users’ queries, major search engines show not only web pages but also other content (e.g., YouTube videos in Google search results). Likewise, the cookpad shows not only recipes but also various other content (e.g., tips, news, and videos related to the user’s queries). For each query, content is selected based on a multi-armed bandit algorithm to maximize its click-through rate.
Research Promotion. To promote food research, cookpad has made its recipes available  through the National Institute of Informatics (NII), a Japanese Research Institute with the goal of advancing informatics research. The 1.7 million recipes involved in this challenge are those that were uploaded to cookpad by the end of September 2014. This data collection was released in February 2015.2 Any researcher in public institutions (e.g., university) can obtain access to the collection for research purposes and as of January 2016, 82 research groups at 56 universities had already done so.
3 Academic Challenges for a Smart Recipe Search
3.1 Cooking Recipe Recommendations Using Surrounding Information
One of the most stressful issues for homemakers is to decide the day’s menu . Many homemakers browse the web to decide what they are going to cook each day. Although most existing recipe search systems request that the user submits a query, such as the name of the menu that they want or names of the ingredients that they want to use [10, 11, 12], people cannot always explain the property of the recipe that they are looking for, even if they have a preference for a recipe. To deal with these issues, we considered that what people want to eat must strongly depend on their daily circumstances and that a recipe cooked in a given situation must be preferred by someone who is in a similar situation. Therefore, we have proposed a method that recommends recipes not only according to a recipe’s properties but also according to the user’s situation.
On the web, there are many blog-type recipes that describe not only the recipe itself but also the reason why the recipe was selected. We analyzed 2,074 blog-type recipes which were randomly collected from “RECIPE BLOG” . We found that 48.2 % blog-type recipes describe at least one reason for selecting recipes and that these reasons could be classified into 18 categories. We then devised an algorithm that extracts situations corresponding to the 18 categories from a user’s life log, such as his or her tweets, and recommends recipes that have similar reasons to the extracted ones. We evaluated the proposed method under three fictitious scenarios; in two of them, the user had vague requirements for recipes and they could only describe their situation as “it was so busy today.” We call it as “situational scenario.” The other had concrete requirements such as “I bought Pacific saury for a good price.” We call this a “procedural scenario.” We input the description of each scenario into the proposed system and obtained the top ten results. As a baseline method, we also made queries derived from each scenario and found ten recipes that contained the queries as a part of the recipes. Five examinees evaluated each recipe recommended by the two methods using a five-point scale. The proposed method obtained a higher score than the baseline method in the two situational scenarios, while the baseline method obtained a higher score in the procedural one. Therefore, the proposed method is useful when a user cannot make his or her requirements clear as mentioned above. For details, please refer to .
3.2 Recipe Comparison Using a Recipe Flow-Graph
4 Assistive Applications for Food Preparation
Figure 3 shows our image of the applications and agendas in this topic. This perspective is inspired by machine translation via an intermediate model; this is, once we map the recipe text and observed food preparation activities onto a semantic model, we can directly compare the text and activities. In this sense, the semantic model of the working process corresponds to an intermediate language model in the machine translation problem.
A typical application is recipe generation from observation, which is achieved by agendas C\(\rightarrow \)B in Fig. 3. There are many challenges to recognizing food preparation activities (agenda C) [18, 19, 20, 21, 22, 23, 24, 25, 26, 27] and some to generating recipe texts (agenda B) .
Recipe translation is realized by A\(\rightarrow \)D\(\rightarrow \)B. Agenda D has not yet been studied in detail; a comparison in the structured recipe representation in Fig. 2 is the first step of D, i.e. this study is front-line research toward this goal. As mentioned in Sect. 3.2, the DAG (or tree) representation of a recipe is a natural structural organization of the cooking process, and some research teams have proposed methods to extract the structure from recipe texts (agenda A) [29, 30, 31]. Most of the challenges facing processes A and B are addressed in the Japanese language [16, 30, 31], and but little in other languages . For recipe translation and localization, it is important to develop these techniques in multiple languages.
Named entity tags.
Action by the chef
Action by foods
State of foods
State of tools
Head verb of a clause for timing, etc.
While the above applications contribute to increasing the variety of recipes, there are a series of proposals to improve recipe presentation [18, 32, 33, 34, 35, 36, 37, 38]. Most of them [32, 33, 34, 35, 37, 38] assume a simple recipe structure, which is a sequence of steps. A few other studies use the DAG or tree structure for a more intelligent recipe presentation [18, 36].
In , we have forecasted the next intended step of a user via human-object interaction on a cooking surface. A human interacts with objects to proceed with cooking tasks. Simultaneously, the human puts objects aside if they are in his or her way. The challenge was to identify the intended step even while such out-of-context interactions are observed with informative interactions. The proposed method estimates the progress of the cooking process from the history of the interactions together with the intended next step. The estimated progress narrows down the options for the next step. In the experiment, the proposed method achieved more than 70 % accuracy in its forecast. Because forecasting is generally a difficult task of pattern recognition, this is a remarkable score, nonetheless the accuracy should be enhanced by more sophisticated semantic models with statistical information.
In this study, we introduced challenges for assisting with everyday cooking activities, particularly menu planning and food preparation. There are many commercial websites that provide recipes. Accurate and easy-to-use recipe search and recommendation tools are important to increase the number of customers. We introduced challenges by cookpad as a practical example. An accurate and large corpus is critical to achieve a practical recipe search engine. Companies pay a lot of efforts to maintain their corpus and keep the system up-to-date.
We also introduced academic challenges for menu planning. There is a lot of room for academic researchers to enhance recipe recommendation. Food preferences are influenced by the user’s day; therefore, a highly customized recipe recommendation requires a daily context of the family members. From this perspective, both web mining and Life logging are within the scope of this application.
While there have been several assistive systems and related methods for food preparation, one of the most important challenges is to obtain a highly informative semantic model of the food preparation process. We believe that matching the recipe text and cooking observations will provide a breakthrough in this topic. Finally, more challenges in non-Japanese languages are waited for cross-cultural applications in everyday cooking.
This work was supported by JSPS KAKENHI Grant Numbers 24240030, 26280039, 26280084.
- 1.Wang, X., Kumar, D., Thome, N., Cord, M., Precioso, F.: Recipe recognition with large multimodal food dataset. In: Proceedings of IEEE International Conference on Multimedia & Expo Workshops, pp. 1–6 (2015)Google Scholar
- 3.Sudo, K., Murasaki, K., Shimamura, J., Taniguchi, Y.: Estimating nutritional value from food images based on semantic segmentation. In: Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct Publication. UbiComp 2014 Adjunct, pp. 571–576 (2014)Google Scholar
- 4.Kitamura, K., Yamasaki, T., Aizawa, K.: Foodlog: capture, analysis and retrieval of personal food images via web. In: Proceedings of the ACM Multimedia 2009 Workshop on Multimedia for Cooking and Eating Activities, pp. 23–30 (2009)Google Scholar
- 5.Khanna, N., Boushey, C.J., Kerr, D., Okos, M., Ebert, D.S., Delp, E.J.: An overview of the technology assisted dietary assessment project at purdue university. In: Proceedings of 2010 IEEE International Symposium on Multimedia, pp. 290–295 (2010)Google Scholar
- 6.Kawano, Y., Yanai, K.: Food image recognition with deep convolutional features. In: Proceedings of ACM UbiComp Workshop on Workshop on Smart Technology for Cooking and Eating Activities (CEA), September 2014Google Scholar
- 7.Takeuchi, T., Fujii, T., Narumi, T., Tanikawa, T., Hirose, M.: Considering individual taste in social feedback to improve eating habits. In: Proceedings of IEEE International Conference on Multimedia & Expo Workshops, pp. 1–6 (2015)Google Scholar
- 8.Harashima, J., Ariga, M., Murata, K., Ioki, M.: A large-scale recipe and meal data collection as infrastructure for food research. In: Proceedings of the 10th International Conference on Language Resources and Evaluation (2016, to appear)Google Scholar
- 9.Mynavi Corporation: Cooking related questionary investi-gation reported by Mynavi woman on 27th (in Japanese). http://woman.mynavi.jp/article/140227-44/. Accessed 1 Feb 2016
- 11.Tsukuda, K., Yamamoto, T., Nakamura, S., Tanaka, K.: Plus one or minus one: a method to browse from an object to another object by adding or deleting an element. In: Bringas, P.G., Hameurlain, A., Quirchmayr, G. (eds.) DEXA 2010, Part II. LNCS, vol. 6262, pp. 258–266. Springer, Heidelberg (2010)CrossRefGoogle Scholar
- 12.Chung, Y.: Finding food entity relationships using user-generated data in recipe service. In: Proceedings of the 21st ACM International Conference on Information and Knowledge Management, pp. 2611–2614 (2012)Google Scholar
- 13.Ai-Land Co., Ltd.: Recipe blog (in Japanese). http://www.recipe-blog.jp/. Accessed 1 Feb 2016
- 14.Kadowaki, T., Mori, S., Yamakata, Y., Tanaka, K.: Recipe search for blog-type recipe articles based on a users situation. In: Proceedings of ACM Conference on Ubiquitous Computing, pp. 497–506 (2014)Google Scholar
- 17.Wang, L., Li, Q., Li, N., Dong, G., Yang, Y.: Substructure similarity measurement in chinese recipes. In: Proceedings of the 17th International Conference on World Wide Web, pp. 979–988 (2008)Google Scholar
- 18.Hashimoto, A., Inoue, J., Funatomi, T., Minoh, M.: How does user’s access to object make HCI smooth in recipe guidance? In: Rau, P.L.P. (ed.) CCD 2014. LNCS, vol. 8528, pp. 150–161. Springer, Heidelberg (2014)Google Scholar
- 20.Iscen, A., Duygulu, P.: Knives are picked before slices are cut: recognition through activity sequence analysis. In: Proceedings of the 5th International Workshop on Multimedia for Cooking and Eating Activities, pp. 3–8 (2013)Google Scholar
- 21.Rohrbach, M., Amin, S., Andriluka, M., Schiele, B.: A database for fine grained activity detection of cooking activities. In: Proceedings of 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1194–1201 (2012)Google Scholar
- 22.Packer, B., Saenko, K., Koller, D.: A combined pose, object, and feature model for action understanding. In: Proceedings of 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1378–1385 (2012)Google Scholar
- 23.Lei, J., Ren, X., Fox, D.: Fine-grained kitchen activity recognition using RGB-D. In: Proceedings of the 2012 ACM Conference on Ubiquitous Computing, pp. 208–211 (2012)Google Scholar
- 24.Hashimoto, A., Inoue, J., Nakamura, K., Funatomi, T., Ueda, M., Yamakata, Y., Minoh, M.: Recognizing ingredients at cutting process by integrating multimodal features. In: Proceedings of the ACM Multimedia 2012 Workshop on Multimedia for Cooking and Eating Activities, pp. 13–18 (2012)Google Scholar
- 25.Ueda, M., Funatomi, T., Hashimoto, A., Watanabe, T., Minoh, M.: Developing a real-time system for measuring the consumption of seasoning. In: Proceedings of IEEE ISM 2011 Workshop on Multimedia for Cooking and Eating Activities, pp. 393–398 (2011)Google Scholar
- 26.Hashimoto, A., Mori, N., Funatomi, T., Mukunoki, M., Kakusho, K., Minoh, M.: Tracking food materials with changing their appearance in food preparing. In: Proceedings of ISM 2010 Workshop on Multimedia for Cooking and Eating Activities, pp. 248–253. IEEE (2010)Google Scholar
- 28.Yamasaki, T., Yoshino, K., Maeta, H., Sasada, T., Hashimoto, A., Funatomi, T., Yamakata, Y., Mori, S.: Procedual text generation from a flow graph. IPSJ J. 57(3) (to appear). Written in JapaneseGoogle Scholar
- 29.Kiddon, C., Ponnuraj, G.T., Zettlemoyer, L., Choi, Y.: Mise en place: unsupervised interpretation of instructional recipes. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 982–992 (2015)Google Scholar
- 30.Maeta, H., Sasada, T., Mori, S.: A framework for procedural text understanding. In: Proceedings of the 14th International Conference on Parsing Technologies (2015)Google Scholar
- 31.Karikome, S., Fujii, A.: Improving structural analysis of cooking recipe text. IEICE Tech. Rep. Data Eng. 112(75), 43–48 (2012)Google Scholar
- 32.Sato, A., Watanabe, K., Rekimoto, J.: Shadow cooking: situated guidance for a fluid cooking experience. In: Stephanidis, C., Antona, M. (eds.) UAHCI 2014, Part III. LNCS, vol. 8515, pp. 558–566. Springer, Heidelberg (2014)Google Scholar
- 33.Matsushima, Y., Funabiki, N., Zhang, Y., Nakanishi, T., Watanabe, K.: Extensions of cooking guidance function on android tablet for homemade cooking assistance system. In: IEEE 2nd Global Conference on Consumer Electronics, pp. 397–401 (2013)Google Scholar
- 35.Uriu, D., Namai, M., Tokuhisa, S., Kashiwagi, R., Inami, M., Okude, N.: Panavi: recipe medium with a sensors-embedded pan for domestic users to master professional culinary arts. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 129–138 (2012)Google Scholar
- 36.Hamada, R., Okabe, J., Ide, I., Sakai, S., Tanaka, H.: Cooking navi: assistant for daily cooking in kitchen. In: Proceedings of the 13th Annual ACM International Conference on Multimedia, pp. 371–374 (2005). Written in JapaneseGoogle Scholar
- 37.Bradbury, J.S., Shell, J.S., Knowles, C.B.: Hands on cooking: towards an attentive kitchen. In: Proceedings of CHI 2003 Extended Abstracts on Human Factors in Computing Systems, pp. 996–997 (2003)Google Scholar
- 38.Ju, W., Hurwitz, R., Judd, T., Lee, B.: Counteractive: an interactive cookbook for the kitchen counter. In: Proceedings of CHI 2001 Extended Abstracts on Human Factors in Computing Systems, pp. 269–270. ACM, New York (2001)Google Scholar