Autonomous Robots

, Volume 43, Issue 6, pp 1393–1418 | Cite as

Semantic and geometric reasoning for robotic grasping: a probabilistic logic approach

  • Laura AntanasEmail author
  • Plinio Moreno
  • Marion Neumann
  • Rui Pimentel de Figueiredo
  • Kristian Kersting
  • José Santos-Victor
  • Luc De Raedt


While any grasp must satisfy the grasping stability criteria, good grasps depend on the specific manipulation scenario: the object, its properties and functionalities, as well as the task and grasp constraints. We propose a probabilistic logic approach for robot grasping, which improves grasping capabilities by leveraging semantic object parts. It provides the robot with semantic reasoning skills about the most likely object part to be grasped, given the task constraints and object properties, while also dealing with the uncertainty of visual perception and grasp planning. The probabilistic logic framework is task-dependent. It semantically reasons about pre-grasp configurations with respect to the intended task and employs object-task affordances and object/task ontologies to encode rules that generalize over similar object parts and object/task categories. The use of probabilistic logic for task-dependent grasping contrasts with current approaches that usually learn direct mappings from visual perceptions to task-dependent grasping points. The logic-based module receives data from a low-level module that extracts semantic objects parts, and sends information to the low-level grasp planner. These three modules define our probabilistic logic framework, which is able to perform robotic grasping in realistic kitchen-related scenarios.


Cognitive robotics Probabilistic logic Task-dependent grasping Semantic grasping Local shape grasping 



  1. Abdo, N., Kretzschmar, H., & Stachniss, C. (2012). From low-level trajectory demonstrations to symbolic actions for planning. In ICAPS workshop on combining task and motion planning for real-world applications (pp. 1–8).Google Scholar
  2. Aleotti, J., & Caselli, S. (2011). Part-based robot grasp planning from human demonstration. In ICRA (pp. 4554–4560).Google Scholar
  3. Antanas, L., Dries, A., Moreno, P., & De Raedt, L. (2018). Relational affordance learning for task-dependent robot grasping. In N. Lachiche & C. Vrain (Eds.), Inductive logic programming (pp. 1–15). Cham: Springer.Google Scholar
  4. ASUS, Asus xtion pro. Accessed 15 July 2014.
  5. Baltzakis, H. Orca simulator. Accessed 15 July 2014.
  6. Barck-Holst, C., Ralph, M., Holmar, F., & Kragic, D. (2009). Learning grasping affordance using probabilistic and ontological approaches. In International conference on advanced robotics (pp. 1–6).Google Scholar
  7. Besl, P. J., & McKay, N. D. (1992). A method for registration of 3-d shapes. TPAMI, 14(2), 239–256.Google Scholar
  8. Bohg, J., Johnson-Roberson, M., León, B., Felip, J., Gratal, X., Bergström, N., et al. (2011). Mind the gap—robotic grasping under incomplete observation. In ICRA (pp. 686–693).Google Scholar
  9. Bohg, J., & Kragic, D. (2010). Learning grasping points with shape context. RAS, 58(4), 362–377.Google Scholar
  10. Bohg, J., Welke, K., Leon, B., Do, M., Song, D., Wohlkinger, W., et al. (2012). Task-based grasp adaptation on a humanoid robot. In IFAC symposium on robot control (pp. 779–786).Google Scholar
  11. Cocora, A., Kersting, K., Plagemann, C., Burgard, W., & Raedt, L. D. (2006). Learning relational navigation policies. In IEEE/RSJ ICIRS (pp. 2792–2797).Google Scholar
  12. Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20, 273–297.zbMATHGoogle Scholar
  13. Dang, H., & Allen, P. K. (2012). Semantic grasping: Planning robotic grasps functionally suitable for an object manipulation task. In IEEE/RSJ ICIRS (pp. 1311–1317).Google Scholar
  14. Detry, R., Ek, C. H., Madry, M., & Kragic, D. (2012a). Compressing grasping experience into a dictionary of prototypical grasp-predicting parts. In International workshop on human-friendly robotics (pp. 1–1).Google Scholar
  15. Detry, R., Ek, C. H., Madry, M., & Kragic, D. (2013). Learning a dictionary of prototypical grasp-predicting parts from grasping experience. In ICRA (pp. 601–608).Google Scholar
  16. Detry, R., Ek, C. H., Madry, M., Piater, J. H., & Kragic, D. (2012b). Generalizing grasps across partly similar objects. In ICRA (pp. 3791–3797).Google Scholar
  17. Fierens, D., Van den Broeck, G., Thon, I., Gutmann, B., & De Raedt, L. (2011). Inference in probabilistic logic programs using weighted CNF’s. In F. Gagliardi Cozman, A. Pfeffer (Eds.) UAI (pp. 211–220).Google Scholar
  18. Figueiredo, R., Moreno, P., & Bernardino, A. (2017). Automatic object shape completion from 3d point clouds for object manipulation. In International joint conference on computer vision, imaging and computer graphics theory and applications (Vol. 4: VISAPP, pp. 565–570).Google Scholar
  19. Fischinger, D., Vincze, M., & Jiang, Y. (2013). Learning grasps for unknown objects in cluttered scenes. In ICRA (pp. 609–616).Google Scholar
  20. Flanagan, J. R., Bowman, M. C., & Johansson, R. S. (2006). Control strategies in object manipulation tasks. Current Opinion in Neurobiology, 16(6), 650–659.Google Scholar
  21. Gibson, J. (1979). The ecological approach to visual perception. Boston: Houghton Mifflin.Google Scholar
  22. Hanheide, M., Gretton, C., Dearden, R., Hawes, N., Wyatt, J. L., Pronobis, A., et al. (2011). Exploiting probabilistic knowledge under uncertain sensing for efficient robot behaviour. In IJCAI (pp. 2442–2449).Google Scholar
  23. Hart, S., Grupen, R. A., & Jensen, D. (2005). A relational representation for procedural task knowledge. In AAAI (pp. 1280–1285).Google Scholar
  24. Jerez, J., & Suero, A. Newton game dynamics. Open-source physics engine. Accessed 15 July 2014.
  25. Jiang, Y., Moseson, S., & Saxena, A. (2011). Efficient grasping from rgbd images: Learning using a new rectangle representation. In ICRA (pp. 3304–3311).Google Scholar
  26. Kroemer, O., Amor, H. B., Ewerton, M., & Peters, J. (2012). Point cloud completion using extrusions. In Humanoids (pp. 680–685).Google Scholar
  27. Kulick, J., Toussaint, M., Lang, T., & Lopes, M. (2013). Active learning for teaching a robot grounded relational symbols. In IJCAI (pp. 1451–1457). AAAI Press.Google Scholar
  28. K. Robotics, Kuka lightweight robot (LWR). Accessed 15 July 2014.
  29. Lang, T., & Toussaint, M. (2010). Planning with noisy probabilistic relational rules. JAIR, 39, 1–49.zbMATHGoogle Scholar
  30. Lenz, I., Lee, H., & Saxena, A. (2015). Deep learning for detecting robotic grasps. CoRR arXiv:1301.3592.
  31. Limketkai, B., Liao, L., & Fox, D. (2005). Relational object maps for mobile robots. In IJCAI (pp. 1471–1476).Google Scholar
  32. Madry, M., Song, D., Ek, C. H., & Kragic, D. (2012a). Robot bring me something to drink from: Object representation for transferring task specific grasps. In ICRA Workshop on semantic perception, mapping and exploration (pp. 1–6).Google Scholar
  33. Madry, M., Song, D., & Kragic, D. (2012b). From object categories to grasp transfer using probabilistic reasoning. In ICRA (pp. 1716–1723).Google Scholar
  34. Marton, Z. C., Rusu, R. B., Jain, D., Klank, U., & Beetz, M. (2009). Probabilistic categorization of kitchen objects in table settings with a composite sensor. In IEEE/RSJ ICIRS (pp. 4777–4784).Google Scholar
  35. Meert, W., Struyf, J., & Blockeel, H. (2008). Learning ground CP-logic theories by leveraging bayesian network learning techniques. Fundamenta Informaticae, 89(1), 131–160.MathSciNetzbMATHGoogle Scholar
  36. Mitra, N. J., Guibas, L., & Pauly, M. (2006). Partial and approximate symmetry detection for 3D geometry. ACM Transactions on Graphics, 25(3), 560–568.Google Scholar
  37. Moldovan, B., Antanas, L., & Hoffmann, M. (2013a). Opening doors: An initial SRL approach. Lecture Notes in Computer Science, 7842, 178–192.MathSciNetGoogle Scholar
  38. Moldovan, B., Moreno, P., Nitti, D., Santos-Victor, J., & De Raedt, L. (2018). Relational affordances for multiple-object manipulation. Autonomous Robots, 42(1), 19–44.Google Scholar
  39. Moldovan, B., Moreno, P., & van Otterlo, M. (2013b) On the use of probabilistic relational affordance models for sequential manipulation tasks in robotics. In ICRA (pp. 1290–1295).Google Scholar
  40. Moldovan, B., Moreno, P., van Otterlo, M., Santos-Victor, J., & De Raedt, L. (2012). Learning relational affordance models for robots in multi-object manipulation tasks. In ICRA (pp. 4373–4378).Google Scholar
  41. Montesano, L., & Lopes, M. (2012). Active learning of visual descriptors for grasping using non-parametric smoothed beta distributions. RAS, 60(3), 452–462.Google Scholar
  42. Muja, M., & Ciocarlie, M. Table top segmentation package. Accessed 15 July 2014.
  43. Neumann, M., Garnett, R., Bauckhage, C., & Kersting, K. (2016). Propagation kernels: Efficient graph kernels from propagated information. Machine Learning, 102(2), 209–245.MathSciNetzbMATHGoogle Scholar
  44. Neumann, M., Moreno, P., Antanas, L., Garnett, R., & Kersting, K. (2013). Graph kernels for object category prediction in task—dependent robot grasping. In MLG (pp. 1–6).Google Scholar
  45. Neumann, M., Patricia, N., Garnett, R., & Kersting, K. (2012). Efficient graph kernels by randomization. In ECML/PKDD (pp. 378–393).Google Scholar
  46. Nyga, D., Balint-Benczedi, F., & Beetz, M. (2014). PR2 Looking at things: Ensemble learning for unstructured information processing with Markov logic networks. In IEEE international conference on robotics and automation (ICRA), Hong Kong, China.Google Scholar
  47. Palinko, O., Sciutti, A., Rea, F., & Sandini, G. (2014). Weight-aware robot motion planning for lift-to-pass action. International conference on human-agent interaction (pp. 193–196). New York, NY, USA: ACM.Google Scholar
  48. Platt, J. C. (1999). Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In A. Smola, P. Bartlett, B. Schölkopf, & D. Schuurmans (Eds.), Advances in large margin classifiers (pp. 61–74). Cambridge: MIT Press.Google Scholar
  49. Rusinkiewicz, S., & Levoy, M. (2001). Efficient variants of the ICP algorithm. In 3DIM (pp. 145–152).Google Scholar
  50. Sánchez, G., & Latombe, J.-C. (2003). A single-query bi-directional probabilistic roadmap planner with lazy collision checking. Robotics research, Volume 6 of Springer tracts in advanced robotics (pp. 403–417). Berlin: Springer.Google Scholar
  51. Saxena, A., Wong, L. L. S., & Ng, A. Y. (2008). Learning grasp strategies with partial shape information. In AAAI (pp. 1491–1494). AAAI Press.Google Scholar
  52. Song, D., Huebner, K., Kyrki, V., & Kragic, D. (2010). Learning task constraints for robot grasping using graphical models. In IEEE/RSJ ICIRS (pp. 1579–1585).Google Scholar
  53. Şucan, I. A., Moll, M., & Kavraki, L. E. (2012). The open motion planning library. Robotics & Automation Magazine, 19(4), 72–82.Google Scholar
  54. Sweeney, J., & Grupen, R. A. (2007). A model of shared grasp affordances from demonstration. In Humanoids (pp. 27–35).Google Scholar
  55. Tenorth, M., & Beetz, M. (2009). Knowrob knowledge processing for autonomous personal robots. In IEEE/RSJ ICIRS (pp. 4261–4266).Google Scholar
  56. Tenorth, M., Profanter, S., Balint-Benczedi, F., & Beetz, M. (2013). Decomposing cad models of objects of daily use and reasoning about their functional parts. In 2013 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 5943–5949). IEEE.Google Scholar
  57. Thrun, S., & Wegbreit, B. (2005). Shape from symmetry. In ICCV (pp. 1824–1831).Google Scholar
  58. Toussaint, M., Plath, N., Lang, T., & Jetchev, N. (2010). Integrated motor control, planning, grasping and high-level reasoning in a blocks world using probabilistic inference. In ICRA (pp. 385–391).Google Scholar
  59. Vennekens, J., Denecker, M., & Bruynooghe, M. (2009). CP-logic: A language of causal probabilistic events and its relation to logic programming. Theory and Practice of Logic Programming, 9(3), 245–308.MathSciNetzbMATHGoogle Scholar
  60. Winkler, J., Bartels, G., Msenlechner, L., & Beetz, M. (2012). Knowledge enabled high-level task abstraction and execution. Conference for Advances in Cognitive Systems, 2(1), 131–148.Google Scholar
  61. W. Robotics. Universal gripper WSG 50. Accessed 15 July 2014.
  62. Yousef, H., Boukallel, M., & Althoefer, K. (2011). Tactile sensing for dexterous in-hand manipulation in roboticsa review. Sensors and Actuators A: Physical, 167(2), 171–187 (Solid-State Sensors. Actuators and Microsystems Workshop).Google Scholar
  63. Zhu, X. (2013). Persistent homology: An introduction and a new text representation for natural language processing. In IJCAI (pp. 1953–1959). AAAI Press.Google Scholar

Copyright information

© Springer Science+Business Media, LLC, part of Springer Nature 2018

Authors and Affiliations

  1. 1.Department of Computer Science, KULeuvenHeverleeBelgium
  2. 2.Institute for Systems and RoboticsLisbonPortugal
  3. 3.Department of Computer Science and EngineeringWashington University in St LouisSt LouisUSA
  4. 4.Computer Science DepartmentTechnical University of DortmundDortmundGermany

Personalised recommendations