Skip to main content

Grasp’D: Differentiable Contact-Rich Grasp Synthesis for Multi-Fingered Hands

  • Conference paper
  • First Online:
Computer Vision – ECCV 2022 (ECCV 2022)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 13666))

Included in the following conference series:

Abstract

The study of hand-object interaction requires generating viable grasp poses for high-dimensional multi-finger models, often relying on analytic grasp synthesis which tends to produce brittle and unnatural results. This paper presents Grasp’D, an approach to grasp synthesis by differentiable contact simulation that can work with both known models and visual inputs. We use gradient-based methods as an alternative to sampling-based grasp synthesis, which fails without simplifying assumptions, such as pre-specified contact locations and eigengrasps. Such assumptions limit grasp discovery and, in particular, exclude high-contact power grasps. In contrast, our simulation-based approach allows for stable, efficient, physically realistic, high-contact grasp synthesis, even for gripper morphologies with high-degrees of freedom. We identify and address challenges in making grasp simulation amenable to gradient-based optimization, such as non-smooth object surface geometry, contact sparsity, and a rugged optimization landscape. Grasp’D compares favorably to analytic grasp synthesis on human and robotic hand models, and resultant grasps achieve over 4\(\,\times \,\) denser contact, leading to significantly higher grasp stability. Video and code available at: graspd-eccv22.github.io.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 89.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 119.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Allshire, A., et al.:Transferring dexterous manipulation from gpu simulation to a remote real-world trifinger. arXiv preprint arXiv:2108.09779 (2021)

  2. Baek, S., Kim, K.I., Kim, T.K.: Pushing the envelope for rgb-based dense 3d hand pose estimation via neural rendering. In: CVPR (2019)

    Google Scholar 

  3. Baydin, A.G., Pearlmutter, B.A., Radul, A.A., Siskind, J.M.: Automatic differentiation in machine learning: a survey. JMLR (2018)

    Google Scholar 

  4. Bender, J., Duriez, C., Jaillet, F., Zachmann, G.: Continuous collision detection between points and signed distance fields. In: Workshop on Virtual Reality Interaction and Physical Simulation (2014)

    Google Scholar 

  5. Bender, J., Erleben, K., Trinkle, J.: Interactive simulation of rigid body dynamics in computer graphics. Comput. Graph. Forum 33(1), 246–270 (2014)

    Google Scholar 

  6. Boukhayma, A., Bem, R.d., Torr, P.H.: 3d hand shape and pose from images in the wild. In: CVPR (2019)

    Google Scholar 

  7. Brahmbhatt, S., Ham, C., Kemp, C.C., Hays, J.: Contactdb: Analyzing and predicting grasp contact via thermal imaging. In: CVPR (2019)

    Google Scholar 

  8. Brahmbhatt, S., Handa, A., Hays, J., Fox, D.: Contactgrasp: Functional multi-finger grasp synthesis from contact. In: IROS (2019)

    Google Scholar 

  9. Brahmbhatt, S., Tang, C., Twigg, C.D., Kemp, C.C., Hays, J.: Contactpose: A dataset of grasps with object contact and hand pose. In: ECCV (2020)

    Google Scholar 

  10. Calli, B., et al.: Yale-cmu-berkeley dataset for robotic manipulation research. Int. J. Robot. Res. 36(3), 027836491770071 (2017)

    Google Scholar 

  11. Chang, A.X., et al.: Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012 (2015)

  12. Chen, T., Xu, J., Agrawal, P.: A system for general in-hand object re-orientation. In: CoRL (2022)

    Google Scholar 

  13. Ciocarlie, M., Goldfeder, C., Allen, P.: Dexterous grasping via eigengrasps: A low-dimensional approach to a high-complexity problem. In: RSS (2007)

    Google Scholar 

  14. Ciocarlie, M.T., Allen, P.K.: Hand posture subspaces for dexterous robotic grasping. Int. J. Robot. Res. 28(7), 851–867 (2009)

    Google Scholar 

  15. Corona, E., Pumarola, A., Alenya, G., Moreno-Noguer, F., Rogez, G.: Ganhand: Predicting human grasp affordances in multi-object scenes. In: CVPR (2020)

    Google Scholar 

  16. Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: CVPR (2005)

    Google Scholar 

  17. Danielczuk, M., Xu, J., Mahler, J., Matl, M., Chentanez, N., Goldberg, K.: Reach: Reducing false negatives in robot grasp planning with a robust efficient area contact hypothesis model. In: International Symposium of Robotic Research (2019)

    Google Scholar 

  18. De Giorgio, A., Romero, M., Onori, M., Wang, L.: Human-machine collaboration in virtual reality for adaptive production engineering. Procedia Manufacturing (2017)

    Google Scholar 

  19. Depierre, A., Dellandréa, E., Chen, L.: Jacquard: A large scale dataset for robotic grasp detection. In: IROS (2018)

    Google Scholar 

  20. Do, T.T., Nguyen, A., Reid, I.: Affordancenet: An end-to-end deep learning approach for object affordance detection. In: ICRA (2018)

    Google Scholar 

  21. Doosti, B., Naha, S., Mirbagheri, M., Crandall, D.J.: Hope-net: A graph-based model for hand-object pose estimation. In: CVPR (2020)

    Google Scholar 

  22. Duan, K., Bai, S., Xie, L., Qi, H., Huang, Q., Tian, Q.: Centernet: Keypoint triplets for object detection. In: ICCV (2019)

    Google Scholar 

  23. Eppner, C., Mousavian, A., Fox, D.: A billion ways to grasp: An evaluation of grasp sampling schemes on a dense, physics-based grasp data set. arXiv preprint arXiv:1912.05604 (2019)

  24. Eppner, C., Mousavian, A., Fox, D.: Acronym: A large-scale grasp dataset based on simulation. In: ICRA (2021)

    Google Scholar 

  25. Fang, K., Zhu, Y., Garg, A., Kurenkov, A., Mehta, V., Fei-Fei, L., Savarese, S.: Learning task-oriented grasping for tool manipulation from simulated self-supervision. Int. J. Robot. Research (IJRR) 39(2–3), 202–216 (2019)

    Google Scholar 

  26. Felzenszwalb, P.F., Girshick, R.B., McAllester, D., Ramanan, D.: Object detection with discriminatively trained part-based models. TPAMI (2009)

    Google Scholar 

  27. Ferrari, C., Canny, J.F.: Planning optimal grasps. In: ICRA (1992)

    Google Scholar 

  28. Freeman, C.D., Frey, E., Raichuk, A., Girgin, S., Mordatch, I., Bachem, O.: Brax - a differentiable physics engine for large scale rigid body simulation (2021). https://github.com/google/brax

  29. Fuhrmann, A., Sobotka, G., Groß, C.: Distance fields for rapid collision detection in physically based modeling. In: Proceedings of GraphiCon (2003)

    Google Scholar 

  30. Gammieri, L., Schumann, M., Pelliccia, L., Di Gironimo, G., Klimant, P.: Coupling of a redundant manipulator with a virtual reality environment to enhance human-robot cooperation. Procedia Cirp (2017)

    Google Scholar 

  31. Garcia-Hernando, G., Yuan, S., Baek, S., Kim, T.K.: First-person hand action benchmark with rgb-d videos and 3d hand pose annotations. In: CVPR (2018)

    Google Scholar 

  32. Ge, L., Ren, Z., Li, Y., Xue, Z., Wang, Y., Cai, J., Yuan, J.: 3d hand shape and pose estimation from a single rgb image. In: CVPR (2019)

    Google Scholar 

  33. Geilinger, M., Hahn, D., Zehnder, J., Bächer, M., Thomaszewski, B., Coros, S.: Add: Analytically differentiable dynamics for multi-body systems with frictional contact. ACM Trans. Graph. 39(6), 1–15 (2020)

    Google Scholar 

  34. Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: CVPR (2014)

    Google Scholar 

  35. Goldfeder, C., Ciocarlie, M., Dang, H., Allen, P.K.: The columbia grasp database. In: ICRA (2009)

    Google Scholar 

  36. Grady, P., Tang, C., Twigg, C.D., Vo, M., Brahmbhatt, S., Kemp, C.C.: Contactopt: Optimizing contact to improve grasps. In: CVPR (2021)

    Google Scholar 

  37. Hamer, H., Schindler, K., Koller-Meier, E., Van Gool, L.: Tracking a hand manipulating an object. In: ICCV (2009)

    Google Scholar 

  38. Hampali, S., Rad, M., Oberweger, M., Lepetit, V.: Honnotate: A method for 3d annotation of hand and object poses. In: CVPR (2020)

    Google Scholar 

  39. Hasson, Y., Varol, G., Tzionas, D., Kalevatykh, I., Black, M.J., Laptev, I., Schmid, C.: Learning joint reconstruction of hands and manipulated objects. In: CVPR (2019)

    Google Scholar 

  40. Heiden, E., Macklin, M., Narang, Y.S., Fox, D., Garg, A., Ramos, F.: DiSECt: A Differentiable Simulation Engine for Autonomous Robotic Cutting. In: RSS (2021)

    Google Scholar 

  41. Heiden, E., Millard, D., Coumans, E., Sheng, Y., Sukhatme, G.S.: NeuralSim: Augmenting differentiable simulators with neural networks. In: ICRA (2021)

    Google Scholar 

  42. Heumer, G., Amor, H.B., Weber, M., Jung, B.: Grasp recognition with uncalibrated data gloves-a comparison of classification methods. In: IEEE Virtual Reality Conference (2007)

    Google Scholar 

  43. Hu, Y., Anderson, L., Li, T.M., Sun, Q., Carr, N., Ragan-Kelley, J., Durand, F.: Difftaichi: Differentiable programming for physical simulation. In: ICLR (2020)

    Google Scholar 

  44. Hu, Y, et al.: Chainqueen: A real-time differentiable physical simulator for soft robotics. In: ICRA (2019)

    Google Scholar 

  45. Huang, D.A., Ma, M., Ma, W.C., Kitani, K.M.: How do we use our hands? discovering a diverse set of common grasps. In: CVPR (2015)

    Google Scholar 

  46. Jiang, H., Liu, S., Wang, J., Wang, X.: Hand-object contact consistency reasoning for human grasps generation. In: ICCV (2021)

    Google Scholar 

  47. Jiang, Z., Zhu, Y., Svetlik, M., Fang, K., Zhu, Y.: Synergies between affordance and geometry: 6-dof grasp detection via implicit representations. arXiv preprint arXiv:2104.01542 (2021)

  48. Kappler, D., Bohg, J., Schaal, S.: Leveraging big data for grasp planning. In: ICRA (2015)

    Google Scholar 

  49. Karunratanakul, K., Yang, J., Zhang, Y., Black, M.J., Muandet, K., Tang, S.: Grasping field: Learning implicit representations for human grasps. In: 3DV (2020)

    Google Scholar 

  50. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: ICLR (2014)

    Google Scholar 

  51. Kokic, M., Kragic, D., Bohg, J.: Learning task-oriented grasping from human activity datasets. IEEE Robotics and Automation Letters (2020)

    Google Scholar 

  52. Lakshmipathy, A., Bauer, D., Bauer, C., Pollard, N.S.: Contact transfer: A direct, user-driven method for human to robot transfer of grasps and manipulations. In: 2022 International Conference on Robotics and Automation (ICRA), pp. 6195–6201. IEEE (2022)

    Google Scholar 

  53. Lau, M., Dev, K., Shi, W., Dorsey, J., Rushmeier, H.: Tactile mesh saliency. ACM Trans. Graph. 35(4), 1–11 (2016)

    Google Scholar 

  54. Liu, M., Pan, Z., Xu, K., Ganguly, K., Manocha, D.: Deep differentiable grasp planner for high-dof grippers. arXiv preprint arXiv:2002.01530 (2020)

  55. Liu, T., Liu, Z., Jiao, Z., Zhu, Y., Zhu, S.C.: Synthesizing diverse and physically stable grasps with arbitrary hand structures using differentiable force closure estimator. IEEE Robotics and Automation Letters (2021)

    Google Scholar 

  56. Lundell, J., et al.: Multi-fingan: Generative coarse-to-fine sampling of multi-finger grasps. arXiv preprint arXiv:2012.09696 (2020)

  57. Lundell, J., Verdoja, F., Kyrki, V.: Ddgc: Generative deep dexterous grasping in clutter. IEEE Robot. Autom. Lett. 6(4), 6599–6906 (2021)

    Google Scholar 

  58. Maas, A.L., Hannun, A.Y., Ng, A.Y., et al.: Rectifier nonlinearities improve neural network acoustic models. In: ICML (2013)

    Google Scholar 

  59. Macklin, M., Erleben, K., Müller, M., Chentanez, N., Jeschke, S., Corse, Z.: Local optimization for robust signed distance field collision. ACM Comput. Graph. Interact. Tech. 3(1), 1–17 (2020)

    Google Scholar 

  60. Mahler, J., et al.: Dex-net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. arXiv preprint arXiv:1703.09312 (2017)

  61. Mahler, J., et al.: Learning ambidextrous robot grasping policies. Sci. Robot. 4(26), eaau4984 (2019)

    Google Scholar 

  62. Mahler, J., Pokorny, F.T., McCarthy, Z., van der Stappen, A.F., Goldberg, K.: Energy-bounded caging: Formal definition and 2-d energy lower bound algorithm based on weighted alpha shapes. IEEE Robotics and Automation Letters (2016)

    Google Scholar 

  63. Miller, A.T., Allen, P.K.: Graspit! a versatile simulator for robotic grasping. IEEE Robot. Autom. Mag. 11(4), 110–122 (2004)

    Google Scholar 

  64. Mordatch, I., Popović, Z., Todorov, E.: Contact-invariant optimization for hand manipulation. In: ACM SIGGRAPH/Eurographics Symposium on Computer Animation (2012)

    Google Scholar 

  65. Mordatch, I., Todorov, E., Popović, Z.: Discovery of complex behaviors through contact-invariant optimization. ACM Trans. Graph. 31(4), 1–8 (2012)

    Google Scholar 

  66. Mousavian, A., Eppner, C., Fox, D.: 6-dof graspnet: Variational grasp generation for object manipulation. In: ICCV (2019)

    Google Scholar 

  67. Nakamura, Y.C., Troniak, D.M., Rodriguez, A., Mason, M.T., Pollard, N.S.: The complexities of grasping in the wild. In: International Conference on Humanoid Robotics (2017)

    Google Scholar 

  68. Osher, S., Fedkiw, R.: Level Set Methods and Dynamic Implicit Surfaces. AMS, vol. 153. Springer, New York (2003). https://doi.org/10.1007/b98879

    Book  MATH  Google Scholar 

  69. Pirk, S., et al.: Understanding and exploiting object interaction landscapes. ACM Trans. Graphi. 36(4), 1 (2017)

    Google Scholar 

  70. Platt, J.C., Barr, A.H.: Constrained differential optimization. In: NeurIPS (1987)

    Google Scholar 

  71. Pokorny, F.T., Kragic, D.: Classical grasp quality evaluation: New algorithms and theory. In: IROS (2013)

    Google Scholar 

  72. Porzi, L., Bulo, S.R., Penate-Sanchez, A., Ricci, E., Moreno-Noguer, F.: Learning depth-aware deep representations for robotic perception. IEEE Robot. Autom. Lett. 2(2), 468–475 (2016)

    Google Scholar 

  73. Qiao, Y.L., Liang, J., Koltun, V., Lin, M.C.: Efficient differentiable simulation of articulated bodies. In: ICML (2021)

    Google Scholar 

  74. Rodriguez, A., Mason, M.T., Ferry, S.: From caging to grasping. International J. Robot. Res. 31(7), 889–900 (2012)

    Google Scholar 

  75. Romero, J., Tzionas, D., Black, M.J.: Embodied hands: Modeling and capturing hands and bodies together. ACM Trans. Graph. 36(6) (2017)

    Google Scholar 

  76. Roy, A., Todorovic, S.: A multi-scale CNN for affordance segmentation in RGB images. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 186–201. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46493-0_12

    Chapter  Google Scholar 

  77. Saxena, A., Driemeyer, J., Kearns, J., Ng, A.: Robotic grasping of novel objects. In: Advances in Neural Information Processing Systems, vol. 19 (2006)

    Google Scholar 

  78. Shao, L., et al.: Unigrasp: Learning a unified model to grasp with multifingered robotic hands. IEEE Robot. Autom. Lett. 5 2286–2293 (2020)

    Google Scholar 

  79. Sundermeyer, M., Mousavian, A., Triebel, R., Fox, D.: Contact-graspnet: Efficient 6-dof grasp generation in cluttered scenes. In: ICRA (2021)

    Google Scholar 

  80. Supančič, J.S., Rogez, G., Yang, Y., Shotton, J., Ramanan, D.: Depth-based hand pose estimation: methods, data, and challenges. IJCV (2018)

    Google Scholar 

  81. Taheri, O., Ghorbani, N., Black, M.J., Tzionas, D.: GRAB: a dataset of whole-body human grasping of objects. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12349, pp. 581–600. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58548-8_34

    Chapter  Google Scholar 

  82. Turpin, D., Wang, L., Tsogkas, S., Dickinson, S., Garg, A.: GIFT: Generalizable Interaction-aware Functional Tool Affordances without Labels. In: Robotics: Systems and Science (RSS) (2021)

    Google Scholar 

  83. Tzionas, D., Ballan, L., Srikantha, A., Aponte, P., Pollefeys, M., Gall, J.: Capturing hands in action using discriminative salient points and physics simulation. IJCV (2016)

    Google Scholar 

  84. Veres, M., Moussa, M., Taylor, G.W.: An integrated simulator and dataset that combines grasping and vision for deep learning. arXiv preprint arXiv:1702.02103 (2017)

  85. Viola, P., Jones, M.: Rapid object detection using a boosted cascade of simple features. In: CVPR (2001)

    Google Scholar 

  86. Weisz, J., Allen, P.K.: Pose error robust grasping from contact wrench space metrics. In: ICRA (2012)

    Google Scholar 

  87. Werling, K., Omens, D., Lee, J., Exarchos, I., Liu, C.K.: Fast and feature-complete differentiable physics for articulated rigid bodies with contact. arXiv preprint arXiv:2103.16021 (2021)

  88. Xu, J., Makoviychuk, V., Narang, Y., Ramos, F., Matusik, W., Garg, A., Macklin, M.: Accelerated Policy Learning with Parallel Differentiable Simulation. In: International Conference on Learning Representations (ICLR) (2022)

    Google Scholar 

  89. Yang, Y., Fermuller, C., Li, Y., Aloimonos, Y.: Grasp type revisited: A modern perspective on a classical feature for vision. In: CVPR (2015)

    Google Scholar 

  90. Zhou, Y., Hauser, K.: 6dof grasp planning by optimizing a deep learning scoring function. In: RSS (2017)

    Google Scholar 

  91. Zimmermann, C., Brox, T.: Learning to estimate 3d hand pose from single rgb images. In: ICCV (2017)

    Google Scholar 

Download references

Acknowledgements

DT was supported in part by a Vector research grant. The authors appreciate the support of NSERC, Vector Institute and Samsung AI. AG was also supported by NSERC Discovery Grant, NSERC Exploration Grant, CIFAR AI Chair, XSeed Discovery Grant from University of Toronto.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Dylan Turpin .

Editor information

Editors and Affiliations

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (mp4 12871 KB)

Supplementary material 2 (pdf 4968 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2022 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Turpin, D. et al. (2022). Grasp’D: Differentiable Contact-Rich Grasp Synthesis for Multi-Fingered Hands. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds) Computer Vision – ECCV 2022. ECCV 2022. Lecture Notes in Computer Science, vol 13666. Springer, Cham. https://doi.org/10.1007/978-3-031-20068-7_12

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-20068-7_12

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-20067-0

  • Online ISBN: 978-3-031-20068-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics