Skip to main content
SpringerLink
Log in

Robotic Cooking Through Pose Extraction from Human Natural Cooking Using OpenPose

  • Conference paper
  • First Online:
Intelligent Autonomous Systems 16 (IAS 2021)

Part of the book series: Lecture Notes in Networks and Systems ((LNNS,volume 412))

Included in the following conference series:

Abstract

Robotic cooking is a difficult task, as the translation from raw recipe instructions to robotic actions involving precise motions and tool-handling is challenging. This paper introduces automated recipe technique recognition based on recording the pose trajectories of human demonstrators carrying out a pancake recipe, as an intuitive method for untrained demonstrators to show a robot how to carry out their recipe variant. A Kinect 2 sensor and the OpenPose neural network are used to extract key timings from the demonstrations, which are replicated when the recipe is carried out by a UR5 arm. Comparing several human-cooked pancakes with their robot-replicated counterparts, the robot’s pancake quality scores were found to be only slightly inferior to the human ones, suggesting that the key parameters selected did encompass the most important variations in cooking technique. Furthermore, we discuss preliminary results in inferring the relationship between the cooking parameters and the quality scores.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 189.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 249.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Bollini, M., Barry, J., Rus, D.: Bakebot: Baking cookies with the pr2. In: The PR2 Workshop: Results, Challenges and Lessons Learned in Advancing Robots with a Common Platform, IROS. 2011

    Google Scholar 

  2. Beetz, M., et al.: Robotic roommates making pancakes. In: 2011 11th IEEE-RAS International Conference on Humanoid Robots. ISSN: 2164–0572, pp. 529–536 (Oct 2011)

    Google Scholar 

  3. Robotics, M.: Moley–The world’s first robotic kitchen. https://www.moley.com (2020). Visited on 05 Dec 2020

  4. Billard, A., Calinon, S., Dillmann, R., Schaal, S.: Handbook of robotics chapter 59: Robot programming by demonstration. In: Handbook of Robotics. Springer (2008)

    Google Scholar 

  5. Caccavale, R., a Saveriano, M., Finzi, A., Lee, D.: Kinesthetic teaching and attentional supervision of structured tasks in human–robot interaction. In: Autonomous Robots, vol. 43.6, pp. 1291–1307 (Aug 2019)

    Google Scholar 

  6. Figueroa, N., Billard, A.: Learning complex manipulation tasks from heterogeneous and unstructured demonstrations. In: IROS Workshop on Synergies between Learning and Interaction, pp. 7 (2017)

    Google Scholar 

  7. Mollard, Y., Munzer, T., Baisero, A., Toussaint, M., Lopes, M.: Robot programming from demonstration, feedback and transfer. In: 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1825–1831 (Sep 2015)

    Google Scholar 

  8. Niekum, S., Osentoski, S., Konidaris, G., Barto, A.G.: Learning and generalization of complex tasks from unstructured demonstrations. In: 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5239–5246. IEEE, Vilamoura-Algarve, Portugal (Oct 2012)

    Google Scholar 

  9. Ramirez-Amaro, K., Beetz, M., Cheng, G.: Transferring skills to humanoid robots by extracting semantic representations from observations of human activities. Artif. Intell. 247, 95–118 (2017)

    Article  MathSciNet  Google Scholar 

  10. Cao, Z., Hidalgo, G., Simon, T., Wei, S.-E., Sheikh, Y.: OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields. In (2018)

    Google Scholar 

  11. Savitzky, A., Golay, M.J.E: Smoothing and differentiation of data by simplified least squares procedures. In: Analytical Chemistry, vol. 36.8, pp. 1627–1639 (1964)

    Google Scholar 

  12. Székely, G.J., Rizzo, M.L., Bakirov, N.K.: Measuring and testing dependence by correlation of distances. In: Ann. Statist. vol. 35.6, pp. 2769–2794 (Dec 2007)

    Google Scholar 

Download references

Acknowledgement

This work was supported by BEKO PLC and Symphony Kitchens, and the Engineering and Physical Sciences Research Council (EPSRC) RoboPatient grant [EP/T00603X/1].

Author information

Authors and Affiliations

  1. Bio-Inspired Robotics Laboratory (BIRL), Department of Engineering, Cambridge University, Cambridge, UK

    Dylan Danno, Simon Hauser & Fumiya Iida

Authors
  1. Dylan Danno
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Simon Hauser
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Fumiya Iida
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Fumiya Iida .

Editor information

Editors and Affiliations

  1. Department of Mechanical Engineering, National University of Singapore, Singapore, Singapore

    Marcelo H. Ang Jr

  2. Department of Precision Engineering, University of Tokyo, Tokyo, Japan

    Hajime Asama

  3. Singapore Institute of Manufacturing Technology, Singapore, Singapore

    Wei Lin

  4. Engineering Product Development, Singapore University of Technology and Design, Singapore, Singapore

    Shaohui Foong

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Danno, D., Hauser, S., Iida, F. (2022). Robotic Cooking Through Pose Extraction from Human Natural Cooking Using OpenPose. In: Ang Jr, M.H., Asama, H., Lin, W., Foong, S. (eds) Intelligent Autonomous Systems 16. IAS 2021. Lecture Notes in Networks and Systems, vol 412. Springer, Cham. https://doi.org/10.1007/978-3-030-95892-3_22

Download citation

Publish with us

Policies and ethics

Search

Navigation