Nonprehensile Manipulation Control and Task Planning for Deformable Object Manipulation: Results from the RoDyMan Project

  • Fabio RuggieroEmail author
  • Jung-Tae Kim
  • Alejandro Gutierrez-Giles
  • Aykut C. Satici
  • Alejandro Donaire
  • Jonathan Cacace
  • Luca Rosario Buonocore
  • Giuseppe Andrea Fontanelli
  • Vincenzo Lippiello
  • Bruno Siciliano
Conference paper
Part of the Lecture Notes in Electrical Engineering book series (LNEE, volume 613)


This chapter aims the broadcasting of the results achieved by the RoDyMan project about the task planning manipulation of deformable objects, and the nonprehensile manipulation control. The final demonstrator of the project is a pizza-making process. After an introduction to the general topic of nonprehensile manipulation, the mechatronic design and the high-level software architecture are described. Then, the smoothed particle hydrodynamic formulation is briefly introduced, along with the description of a detection method for a deformable object. The task planning for stretching a modelling clay, emulating the pizza dough, is sketched. After, the problematic control objective is split into several nonprehensile motion primitives: holonomic and nonholonomic rolling, friction-induced manipulation, and tossing are the described primitives. This chapter highlights the achievements reached so far by the project, and pave the way towards future research directions.


Nonprehensile dynamic manipulation Deformable object Perception Robot planning Robot control 



The research leading to these results has been supported by the RoDyMan project, which has received funding from the European Research Council FP7 Ideas under Advanced Grant agreement number 320992.


  1. 1.
    Amjid MR, Shehzad A, Hussain S, Shabbir MA, Khan MR, Shoaib M (2013) A comprehensive review on wheat flour dough rheology. Pak J Food Sci 23:105–123Google Scholar
  2. 2.
    Anshelevich E, Owens S, Lamiraux F, Kavraki LE (2000) Deformable volumes in path planning applications. In: 2000 IEEE international conference on robotics and automation, San Francisco, CA, pp 2290–2285Google Scholar
  3. 3.
    Bätz G, Mettin U, Schimdts A, Scheint M, Wollherr D, Shiriaev A (2010) Ball dribbling with an underactuated continuous-time control phase: theory & experiments. In: 2010 IEEE/RSJ international conference on intelligent robots and systems, Taipei, TW, pp 2890–2895Google Scholar
  4. 4.
    Bätz G, Yaqub A, Wu H, Kuhnlenz K, Wollherr D, Buss M (2010) Dnamic manipulation: nonprehensile ball catching. In: 18th mediterranean conference on control and automation, Marrakech, MA, pp 365–370Google Scholar
  5. 5.
    Bay H, Ess A, Tuytelaars T, Van Gool L (2006) Surf: speeded up robust features. In: European conference on computer vision, Graz, AT, pp 404–417Google Scholar
  6. 6.
    Bender J, Koschier D (2015) Divergence-free smoothed particle hydrodynamics. In: 14th ACM SIGGRAPH/eurographics symposium on computer animation, Los Angeles, CA, USA, pp 147–155Google Scholar
  7. 7.
    Bender J, Koschier D (2017) Divergence-free SPH for incompressible and viscous fluids. IEEE Trans Vis Comput Graph 23:1193–1206CrossRefGoogle Scholar
  8. 8.
    Bittanti S, Laub A, Willems J (1991) The Riccati equation. Springer, New YorkzbMATHCrossRefGoogle Scholar
  9. 9.
    Bloch AM, Crouch PE (1995) Nonholonomic control systems on riemannian manifolds. SIAM J Control Optim 33(1):126–148MathSciNetzbMATHCrossRefGoogle Scholar
  10. 10.
    Boothby WM (1986) An introduction to differentiable manifolds and Riemannian geometry, vol 120. Academic PressGoogle Scholar
  11. 11.
    Cacace J, Finzi A, Lippiello V, Loianno G, Sanzone D (2015) Aerial service vehicles for industrial inspection: task decomposition and plan execution. Appl Intell 42:49–62CrossRefGoogle Scholar
  12. 12.
    Canny J (1986) A computational approach to edge detection. IEEE Trans Pattern Anal Mach Intell PAMI 8(6):679–698CrossRefGoogle Scholar
  13. 13.
    Chitta S, Sucan I, Cousins S (2012) Moveit! [ros topics]. IEEE Robot Autom Mag 19:18–19CrossRefGoogle Scholar
  14. 14.
    Chung TJ (2010) Computational fluid dynamics, 2nd edn. Cambridge University Press, CambridgezbMATHCrossRefGoogle Scholar
  15. 15.
    Cretu AM, Payeur P, Petriu E (2008) Neural network mapping and clustering of elastic behavior from tactile and range imaging for virtualized reality applications. IEEE Trans Instrum Meas 57(9):1918–1928CrossRefGoogle Scholar
  16. 16.
    Donaire A, Ruggiero F, Buonocore L, Lippiello V, Siciliano B (2016) Passivity-based control for a rolling-balancing system: the nonprehensile disk-on-disk. IEEE Trans Control Syst Technol 25(6):2135–2142CrossRefGoogle Scholar
  17. 17.
    Eymard R, Gallouët TR, Herbin R (2000) The finite volume method. In: Handbook of numerical analysis, vol. 7, pp 713–1020Google Scholar
  18. 18.
    Frank B, Schmedding R, Stachniss C, Teschner M, Burgard W (2010) Learning the elasticity parameters of deformable objects with a manipulation robot. In: 2010 IEEE/RSJ international conference on intelligent robots and systems, Taipei, TW, pp 1877–1883Google Scholar
  19. 19.
    Frank B, Stachniss C, Abdo N, Burgard W (2011) Efficient motion planning for manipulation robots in environments with deformable objects. In: 2011 IEEE/RSJ international conference on intelligent robots and systems, San Francisco, CA, USA, pp 2180–2185Google Scholar
  20. 20.
    Frisken S, Gibson F, Mirtich B (1997) A survey of deformable modeling in computer graphics. Techincal report, Brigham and Women’s HospitalGoogle Scholar
  21. 21.
    Gonzalez RC, Woods RE (2006) Digital image processing, 3rd edn. Prentice-Hall Inc., Upper Saddle RiverGoogle Scholar
  22. 22.
    Gutiérrez-Giles A, Ruggiero F, Lippiello V, Siciliano B (2017) Modelling and control of a robotic hula-hoop system without velocity measurements. In: 20th world congress of the international federation of automatic control, Toulouse, FR, pp 9808–9814Google Scholar
  23. 23.
    Gutiérrez-Giles A, Ruggiero F, Lippiello V, Siciliano B (2018) Nonprehensile manipulation of an underactuated mechanical system with second-order nonholonomic constraints: the robotic hula-hoop. IEEE Robot Autom Lett 3(2):1136–1143CrossRefGoogle Scholar
  24. 24.
    Harris C, Stephens M (1988) A combined corner and edge detector. In: 4th Alvey vision conference, Manchester, UK, pp 147–151Google Scholar
  25. 25.
    Hartley R, Zisserman A (2004) Multiple view geometry in computer vision, 2nd edn. Cambridge University Press, CambridgezbMATHCrossRefGoogle Scholar
  26. 26.
    Higashimori M, Utsumi K, Kaneko M (2008) Dexterous hyper plate inspired by pizza manipulation. In: 2008 IEEE international conference on robotics and automation, Pasadena, CA, USA, pp 399–406Google Scholar
  27. 27.
    Higashimori M, Utsumi K, Omoto Y, Kaneko M (2009) Dynamic manipulation inspired by the handling of a pizza peel. IEEE Trans. Robot. 25(4):829–838CrossRefGoogle Scholar
  28. 28.
    Higashimori M, Yoshimoto K, Kaneko M (2010) Active shaping of an unknown rheological object based on deformation decomposition into elasticity and plasticity. In: 2010 IEEE international conference on robotics and automation, Anchorage, AK, USA, pp 5120–5126Google Scholar
  29. 29.
    Irving G, Teran J, Fedkiw R (2004) Invertible finite elements for robust simulation of large deformation. In: 2004 ACM SIGGRAPH/eurographics symposium on computer animation, Grenoble, FR, pp 131–140Google Scholar
  30. 30.
    Kato H, Billinghurst M (1999) Marker tracking and HMD calibration for a video-based augmented reality conferencing system. In: 2nd IEEE and ACM international workshop on augmented reality, San Francisco, CA, USA, pp 85–94Google Scholar
  31. 31.
    Lang J (2004) An acquisition method for interactive deformable models. In: Second international conference on creating, connecting and collaborating through computing, Kyoto, JP, pp 165–170Google Scholar
  32. 32.
    Lippiello V, Ruggiero F, Siciliano B (2016) The effects of shapes in input-state linearization for stabilization of nonprehensile planar rolling dynamic manipulation. Robot Autom Lett 1(1):492–499CrossRefGoogle Scholar
  33. 33.
    Liu GR, Zhang J, Lam KY, Li H, Xu G, Zhong ZH, Li GY, Han X (2008) A gradient smoothing method (GSM) with directional correction for solid mechanics problems. Comput Mech 41(3):457–472zbMATHCrossRefGoogle Scholar
  34. 34.
    Liu MB, Liu GR (2010) Smoothed particle hydrodynamics (SPH): An overview and recent developments. Arch. Comput. Methods in Eng. 17(1):25–76MathSciNetzbMATHCrossRefGoogle Scholar
  35. 35.
    Lowe DG (1999) Object recognition from local scale-invariant features. In: International conference on computer vision, Kerkyra, GR, vol 2, pp 1150–1157Google Scholar
  36. 36.
    Lynch KM, Mason MT (1999) Dynamic nonprehensile manipulation: controllability, planning, and experiments. Int J Robot. Res. 18(1):64–92CrossRefGoogle Scholar
  37. 37.
    Lynch KM, Murphey TD (2003) Control of nonprehensile manipulation. In: Bicchi A, Prattichizzo D, Christensen H (eds) Control problems in robotics, vol 4. Springer tracts in advanced robotics. Springer, Heidelberg, pp 39–57zbMATHCrossRefGoogle Scholar
  38. 38.
    Mitsoulis E (2008) Numerical simulation of calendering viscoplastic fluids. J Non-Newton Fluid Mech 154:77–88zbMATHCrossRefGoogle Scholar
  39. 39.
    Mitsoulis E, Hatzikiriakos SG (2009) Rolling of bread dough: experiments and simulations. Food Bioprod Process 87(2):124–138CrossRefGoogle Scholar
  40. 40.
    Monaghan JJ (1992) Smoothed particle hydrodynamics. Annu. Rev. Astron. Astrophys. 30:543–574CrossRefGoogle Scholar
  41. 41.
    Monaghan JJ (1994) Simulating free surface flows with SPH. J. Comput. Phys. 110(2):399–406zbMATHCrossRefGoogle Scholar
  42. 42.
    Monaghan JJ (2005) Smoothed particle hydrodynamics. Rep Progress Phys 68:1703–1759MathSciNetzbMATHCrossRefGoogle Scholar
  43. 43.
    Murray RM, Li Z, Sastry S (1994) A mathematical introduction to robotic manipulation. CRC PressGoogle Scholar
  44. 44.
    Nealen A, Müller M, Keiser R, Boxerman R, Carlson M (2005) Physically based deformable models in computer graphics. Eurographics: State of the Art Report, pp 71–94Google Scholar
  45. 45.
    Peer A, Ihmsen M, Cornelis J, Teschner M (2015) An implicit viscosity formulation for SPH fluids. ACM Trans Graph 34(4):1–10CrossRefGoogle Scholar
  46. 46.
    Petit A, Lippiello V, Siciliano B (2015) Real-time tracking of 3D elastic objects with an RGB-D sensor. In: 2015 IEEE/RSJ international conference on intelligent robots and systems, Hamburg, GE, pp. 3914–3921Google Scholar
  47. 47.
    Phillips-Grafflin C, Berenson D (2014) A representation of deformable objects for motion planning with no physical simulation. In: 2014 IEEE International conference on robotics and automation, Hong Kong, CN, pp 98–105Google Scholar
  48. 48.
    Prattichizzo D, Trinkle J (2016) Grasping. In: Siciliano B, Khatib O (eds) Springer handbook of robotics, pp 955–988. SpringerGoogle Scholar
  49. 49.
    Reist P, D’Andrea R (2012) Design and analysis of a blind juggling robot. IEEE Trans Robot 28(6):1228–1243CrossRefGoogle Scholar
  50. 50.
    Reznik DS, Canny JF (2001) C’mon part, do the local motion! In: 2001 IEEE international conference on robotics and automation, vol 3, Seul, KR, pp 2235–2242Google Scholar
  51. 51.
    Rosten E, Drummond T(2006) Machine learning for high-speed corner detection. In: European conference on computer vision, Graz, AT, pp 430–443Google Scholar
  52. 52.
    Ruggiero F, Lippiello V, Siciliano B (2018) Nonprehensile dynamic manipulation: a survey. IEEE Robot Autom Lett 3(3):1711–1718CrossRefGoogle Scholar
  53. 53.
    Satici A, Ruggiero F, Lippiello V, Siciliano B (2016) A coordinate-free framework for robotic pizza tossing and catching. In: 2016 IEEE international conference on robotics and automation, Stockholm, SE, pp 3932–3939Google Scholar
  54. 54.
    Serra D, Ferguson J, Ruggiero F, Siniscalco A, Petit A, Lippiello V, Siciliano B (2018) On the experiments about the nonprehensile reconfiguration of a rolling sphere on a plate. In: 26th mediterranean conference on control and automation, Zadar, HRGoogle Scholar
  55. 55.
    Serra D, Ruggiero F, Donaire A, Buonocore L, Lippiello V, Siciliano B (2019) Control of nonprehensile planar rolling manipulation: a passivity based approach. IEEE Trans Robot 35(2):317–329. Scholar
  56. 56.
    Serra D, Satici A, Ruggiero F, Lippiello V, Siciliano B (2016) An optimal trajectory planner for a robotic batting task: the table tennis example. In: 13th international conference on informatics in control, automation and robotics, Lisbon, PT, pp 90–101 (2016)Google Scholar
  57. 57.
    Shao S, Lo E (2003) Incompressible SPH method for simulating newtonian and non-newtonian flows with a free surface. Adv Water Resour 26(7):787–800CrossRefGoogle Scholar
  58. 58.
    Shiriaev A, Perram J, Canudas-de Wit C (2005) Constructive tool for orbital stabilization of underactuated nonlinear systems: virtual constraints approach. IEEE Trans Autom Control 50(8):1164–1176MathSciNetzbMATHCrossRefGoogle Scholar
  59. 59.
    Sofou S, Muliawan EB, Hatzikiriakos SG, Mitsoulis E (2008) Rheological characterization and constitutive modeling of bread dough. Rheologica Acta 47(4):369–381CrossRefGoogle Scholar
  60. 60.
    Sousa CD, Cortesão R (2014) Physically feasibility of robot base inertial parameters identification: a linear matrix inequality approach. Int J Robot Res 33(6):931–944CrossRefGoogle Scholar
  61. 61.
    de Souza Andrade LF, Sandim M, Petronetto F, Pagliosa P, Paiva A (2014) SPH fluids for viscous jet buckling. In: 27th conference on graphics, patterns and images, Rio de Janeiro, BR, pp 65–72Google Scholar
  62. 62.
    de Souza Andrade LF, Sandim M, Petronetto F, Pagliosa P, Paiva A (2015) Particle-based fluids for viscous jet buckling. Comput Graph 52:106–115CrossRefGoogle Scholar
  63. 63.
    Spong MW (1994) Partial feedback linearization of underactuated mechanical systems. In: IEEE/RSJ/GI international conference on intelligent robots and systems, Munich, DE, pp 314–321Google Scholar
  64. 64.
    Takahashi T, Dobashi Y, Fujishiro I, Nishita T, Lin MC (2015) Implicit formulation for SPH-based viscous fluids. Comput Graph Forum 34(2):493–502CrossRefGoogle Scholar
  65. 65.
    Tokumoto S, Hirai S (2002) Deformation control of rheological food dough using a forming process model. In: 2002 IEEE international conference on robotics and automation, Washington, DC, USA, pp 1457–1464Google Scholar
  66. 66.
    Van Bockstaele F, De Leyn I, Eeckhout M, Dewettinck K (2008) Rheological properties of wheat flour dough and the relationship with bread volume. i. creep-recovery measurements. Cereal Chem J 85(6):753CrossRefGoogle Scholar
  67. 67.
    Vose TH, Umbanhowar P, Lynch KM (2009) Friction-induced lines of attraction and repulsion for parts sliding on an oscillated plate. IEEE Trans Autom Sci Eng 6(4):685–699CrossRefGoogle Scholar
  68. 68.
    Vose TH, Umbanhowar P, Lynch KM (2012) Sliding manipulation of rigid bodies on a controlled 6-dof plate. Int J Robot Res 31(7):819–838CrossRefGoogle Scholar
  69. 69.
    Vose T, Umbanhowar P, Lynch K (2009) Friction-induced velocity fields for point parts sliding on a rigid oscillated plate. Int J Robot Res 28(8):1020–1039CrossRefGoogle Scholar
  70. 70.
    Wada T, Hirai S, Kawamura S, Kamiji N (2001) Robust manipulation of deformable objects by a simple PID feedback. In: 2001 IEEE international conference on robotics and automation, Seoul, KR, pp 85–90Google Scholar
  71. 71.
    Wieser H (2007) Chemistry of gluten proteins. Food Microbiol 24(2):115–119CrossRefGoogle Scholar
  72. 72.
    Woodruff J, Lynch K (2017) Planning and control for dynamic, nonprehensile, and hybrid manipulation tasks. In: 2017 IEEE international conference on robotics and automation, Singapore, pp 4066–4073Google Scholar
  73. 73.
    Zienkiewicz O, Taylor R, Zhu J (2005) The finite element method set, 6th edn. Butterworth-Heinemann, OxfordGoogle Scholar

Copyright information

© Springer Nature Switzerland AG 2020

Authors and Affiliations

  • Fabio Ruggiero
    • 1
    Email author
  • Jung-Tae Kim
    • 1
  • Alejandro Gutierrez-Giles
    • 2
  • Aykut C. Satici
    • 3
  • Alejandro Donaire
    • 4
  • Jonathan Cacace
    • 1
  • Luca Rosario Buonocore
    • 5
  • Giuseppe Andrea Fontanelli
    • 1
  • Vincenzo Lippiello
    • 1
  • Bruno Siciliano
    • 1
  1. 1.CREATE Consortium and Department of Electrical Engineering and Information TechnologyUniversity of Naples Federico IINaplesItaly
  2. 2.Center for Research and Advanced Studies of the National Polytechnic InstituteMexico CityMexico
  3. 3.Mechanical and Biomedical EngineeringBoise State UniversityBoiseUSA
  4. 4.School of EngineeringThe University of NewcastleCallaghanAustralia
  5. 5.CERNGeneve 23Switzerland

Personalised recommendations