Skip to main content
Log in

Haptic-guided grasping to minimise torque effort during robotic telemanipulation

  • Published:
Autonomous Robots Aims and scope Submit manuscript

Abstract

Teleoperating robotic manipulators can be complicated and cognitively demanding for the human operator. Despite these difficulties, teleoperated robotic systems are still popular in several industrial applications, e.g., remote handling of hazardous material. In this context, we present a novel haptic shared control method for minimising the manipulator torque effort during remote manipulative actions in which an operator is assisted in selecting a suitable grasping pose for then displacing an object along a desired trajectory. Minimising torque is important because it reduces the system operating cost and extends the range of objects that can be manipulated. We demonstrate the effectiveness of the proposed approach in a series of representative real-world pick-and-place experiments as well as in a human subjects study. The reported results prove the effectiveness of our shared control vs. a standard teleoperation approach. We also find that haptic-only guidance performs better than visual-only guidance, although combining them together leads to the best overall results.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

References

  • Abbeel, P., Coates, A., & Ng, A. Y. (2010). Autonomous helicopter aerobatics through apprenticeship learning. The International Journal of Robotics Research, 29(13), 1608–1639.

    Article  Google Scholar 

  • Abbink, D. A., Mulder, M., & Boer, E. R. (2012). Haptic shared control: Smoothly shifting control authority? Cognition, Technology & Work, 14(1), 19–28.

    Article  Google Scholar 

  • Abi-Farraj, F., Henze, B., Werner, A., Panzirsch, M., Ott, C. & Roa, M. A. (2018). Humanoid teleoperation using task-relevant haptic feedback. In 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 5010–5017). IEEE.

  • Abi-Farraj, F., Pedemonte, N. & Giordano P. R. (2016). A visual-based shared control architecture for remote telemanipulation. In 2016 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 4266–4273). IEEE.

  • Abi-Farraj, F., Pacchierotti, C., Arenz, O., Neumann, G., & Robuffo Giordano, P. (2019). A haptic shared-control architecture for guided multi-target robotic grasping. IEEE Transactions on Haptics, 13(2), 270–285.

    Article  Google Scholar 

  • Achibet, M., Marchal, M., Argelaguet, F., & Lécuyer, A. (2014). The virtual mitten: A novel interaction paradigm for visuo-haptic manipulation of objects using grip force. In IEEE symposium on 3D user interfaces (3DUI) (pp. 59–66).

  • Adjigble, M., Marturi, N., Ortenzi, V. & Stolkin, R. (2019). An assisted telemanipulation approach: combining autonomous grasp planning with haptic cues. In 2019 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 3164–3171). IEEE.

  • Baronti, L., Alston, M., Mavrakis, N., Ghalamzan, E., Amir, M., Castellani, M., et al. (2019). Primitive shape fitting in point clouds using the bees algorithm. Applied Sciences, 9(23), 5198.

    Article  Google Scholar 

  • Boessenkool, H., Abbink, D. A., Heemskerk, C. J. M., van der Helm, F. C. T., & Wildenbeest, J. G. W. (2013). A task-specific analysis of the benefit of haptic shared control during telemanipulation. Transactions on Haptics, 6(1), 2–12.

    Article  Google Scholar 

  • Bogue, R. (2011). Robots in the nuclear industry: A review of technologies and applications. Industrial Robot: An International Journal, 38(2), 113–118.

    Article  Google Scholar 

  • Bruemmer, D. J., Few, D. A., Boring, R. L., Marble, J. L., Walton, M. C., & Nielsen, C. W. (2005). Shared understanding for collaborative control. IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, 35(4), 494–504.

    Article  Google Scholar 

  • Cipriani, C., Zaccone, F., Micera, S., & Carrozza, M. C. (2008). On the shared control of an EMG-controlled prosthetic hand: Analysis of user prosthesis interaction. IEEE Transactions on Robotics, 24(1), 170–184.

    Article  Google Scholar 

  • Cognetti, M., Aggravi, M., Pacchierotti, C., Salaris, P., & Giordano, P. R. (2020). Perception-aware human-assisted navigation of mobile robots on persistent trajectories. IEEE Robotics and Automation Letters, 5(3), 4711–4718.

    Article  Google Scholar 

  • Corucci, F. & Ruffaldi, E. (2016). Toward autonomous robots for demolitions in unstructured environments. In Intelligent autonomous systems (Vol. 13, pp. 1515–1532). Springer.

  • Detry, R., Papon, J. & Matthies, L. (2017). Task-oriented grasping with semantic and geometric scene understanding. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 3266–3273). IEEE.

  • Duchaine, V., & Gosselin, C. M. (2007). General model of human-robot cooperation using a novel velocity based variable impedance control. In 2007 IEEE world haptics conference (pp. 446–451).

  • Franchi, A., Secchi, C., Ryll, M., Bulthoff, H. H., & Robuffo Giordano, P. (2012). Shared control: Balancing autonomy and human assistance with a group of quadrotor UAVs. IEEE Robotics & Automation Magazine, 19(3), 57–68.

    Article  Google Scholar 

  • Ghalamzan, E. A. M., Abi-Farraj, F., Giordano, P. R. & Stolkin, R. (2017). Human-in-the-loop optimisation: Mixed initiative grasping for optimally facilitating post-grasp manipulative actions. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 3386–3393). IEEE.

  • Ghalamzan E, AM., Mavrakis, N., Kopicki, M., Stolkin, R., Leonardis, A., et al. (2016). Task-relevant grasp selection: A joint solution to planning grasps and manipulative motion trajectories. In 2016 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 907–914). IEEE.

  • Horowitz, M. B. & Burdick, J. W. (2012). Combined grasp and manipulation planning as a trajectory optimization problem. In International conference on robotics and automation (pp. 584–591). IEEE.

  • Izadi, V., Bhardwaj, A. & Ghasemi, A. H. (2020). Impedance modulation for negotiating control authority in a haptic shared control paradigm. In 2020 America control conference (ACC) (pp. 2478–2483). IEEE.

  • James, S., Wohlhart, P., Kalakrishnan, M., Kalashnikov, D., Irpan, A., Ibarz, J., Levine, S., Hadsell, R. & Bousmalis, K. (2019). Sim-to-real via sim-to-sim: Data-efficient robotic grasping via randomized-to-canonical adaptation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 12627–12637).

  • Kopicki, M. S., Belter, D., & Wyatt, J. L. (2019). Learning better generative models for dexterous, single-view grasping of novel objects. The International Journal of Robotics Research, 38(10–11), 1246–1267.

    Article  Google Scholar 

  • Kopicki, M., Detry, R., Adjigble, M., Stolkin, R., Leonardis, A., & Wyatt, J. L. (2016). One-shot learning and generation of dexterous grasps for novel objects. The International Journal of Robotics Research, 35(8), 959–976.

    Article  Google Scholar 

  • Kormushev, P., Calinon, S., & Caldwell, D. G. (2011). Imitation learning of positional and force skills demonstrated via kinesthetic teaching and haptic input. Advanced Robotics, 25(5), 581–603.

    Article  Google Scholar 

  • Levine, S., Pastor, P., Krizhevsky, A., Ibarz, J., & Quillen, D. (2018). Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. The International Journal of Robotics Research, 37(4–5), 421–436.

    Article  Google Scholar 

  • Ly, K. T., Poozhiyil, M., Pandya, H., Neumann, G. & Kucukyilmaz, A. (2021). Intent-aware predictive haptic guidance and its application to shared control teleoperation. In 2021 30th IEEE international conference on robot & human interactive communication (RO-MAN) (pp. 565–572). IEEE.

  • Maisto, M., Pacchierotti, C., Chinello, F., Salvietti, G., De Luca, A., & Prattichizzo, D. (2017). Evaluation of wearable haptic systems for the fingers in augmented reality applications. IEEE Transactions on Haptics, 10(4), 511–522.

    Article  Google Scholar 

  • Masone, C., Giordano, P. R., Bülthoff, H. H. & Franchi, A. (2014). Semi-autonomous trajectory generation for mobile robots with integral haptic shared control. In 2014 international conference on robotics and automation (ICRA) (pp. 6468–6475).

  • Mavrakis, N., Ghalamzan, E. A. M. & Stolkin, R. (2017). Safe robotic grasping: Minimum impact-force grasp selection. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 4034–4041). IEEE.

  • Mavrakis, N., Ghalamzan E, AM., Stolkin, R., Baronti, L., Kopicki, M., Castellani, M., et al. (2016). Analysis of the inertia and dynamics of grasped objects, for choosing optimal grasps to enable torque-efficient post-grasp manipulations. In 2016 IEEE-RAS international conference on humanoid robots (Humanoids) (pp. 171–178). IEEE.

  • Mavrakis, N., E, AMG., Stolkin, R. (2020). Estimating an object’s inertial parameters by robotic pushing: A data-driven approach. In 2020 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 1–9). IEEE.

  • Meli, L., Pacchierotti, C., & Prattichizzo, D. (2017). Experimental evaluation of magnified haptic feedback for robot-assisted needle insertion and palpation. The International Journal of Medical Robotics and Computer Assisted Surgery, 13(4), e1809.

    Article  Google Scholar 

  • Meli, L., Pacchierotti, C., Salvietti, G., Chinello, F., Maisto, M., De Luca, A., & Prattichizzo, D. (2018). Combining wearable finger haptics and augmented reality: User evaluation using an external camera and the microsoft hololens. IEEE Robotics and Automation Letters, 3(4), 4297–4304.

    Article  Google Scholar 

  • Miller, A. T., & Allen, P. K. (2004). Graspit! a versatile simulator for robotic grasping. IEEE Robotics & Automation Magazine, 11(4), 110–122.

    Article  Google Scholar 

  • Murray, R. M., Li, Z., Sastry, S. S., & Sastry, S. S. (1994). A mathematical introduction to robotic manipulation. CRC Press.

    MATH  Google Scholar 

  • Nelson, B., Morrow, J., & Khosla, P. (1996). Robotic manipulation using high bandwidth force and vision feedback. Mathematical and Computer Modelling, 24(5), 11–29.

    Article  MATH  Google Scholar 

  • Pacchierotti, C. (2015). Cutaneous haptic feedback in robotic teleoperation. Springer.

    Book  Google Scholar 

  • Pacchierotti, C., Ongaro, F., Van den Brink, F., Yoon, C., Prattichizzo, D., Gracias, D. H., & Misra, S. (2018). Steering and control of miniaturized untethered soft magnetic grippers with haptic assistance. IEEE Transactions on Automation Science and Engineering, 15(1), 290–306.

    Article  Google Scholar 

  • Pacchierotti, C., Prattichizzo, D., & Kuchenbecker, K. J. (2015). Cutaneous feedback of fingertip deformation and vibration for palpation in robotic surgery. IEEE Transactions on Biomedical Engineering, 63(2), 278–287.

    Article  Google Scholar 

  • Panesar, S., Cagle, Y., Chander, D., Morey, J., Fernandez-Miranda, J., & Kliot, M. (2019). Artificial intelligence and the future of surgical robotics. Annals of Surgery, 270(2), 223–226.

    Article  Google Scholar 

  • Pardi, T., Stolkin, R., et al. (2018). Choosing grasps to enable collision-free post-grasp manipulations. In 2018 IEEE-RAS international conference on humanoid robots (Humanoids). (pp. 299–305). IEEE.

  • Pardi, T., Ortenzi, V., Fairbairn, C., Pipe, T., Esfahani, A. M. G., & Stolkin, R. (2020). Planning maximum-manipulability cutting paths. IEEE Robotics and Automation Letters, 5(2), 1999–2006.

    Article  Google Scholar 

  • Parsa, S., Kamale, D., Mghames, S., Nazari, K., Pardi, T., Srinivasan, A. R., Neumann, G., Hanheide, M. & Amir, G. E. (2020). Haptic-guided shared control grasping: Collision-free manipulation. In 2020 IEEE 16th international conference on automation science and engineering (CASE) (pp. 1552–1557). IEEE.

  • Parsa, S., Maior, H., Thumwood, A., Wilson, M., Hanheide, M., & Amir, G.E. (2022). The impact of motion scaling and haptic guidance on operators’ workload and performance in teleoperation. In Proceedings o of conference on human factors in computing systems p accepted.

  • Pearce, F. (2015). Shocking state of world’s riskiest nuclear waste site. https://www.newscientist.com.

  • Peternel, L., Fang, C., Laghi, M., Bicchi, A., Tsagarakis, N. & Ajoudani, A. (2020). Human arm posture optimisation in bilateral teleoperation through interface reconfiguration. In 2020 8th IEEE RAS/EMBS international conference for biomedical robotics and biomechatronics (BioRob) (pp. 1102–1108). IEEE.

  • Peternel, L., Petrič, T., & Babič, J. (2018). Robotic assembly solution by human-in-the-loop teaching method based on real-time stiffness modulation. Autonomous Robots, 42(1), 1–17.

    Article  Google Scholar 

  • Rahal, R., Abi-Farraj, F., Giordano, P. & Pacchierotti, C. (2019). Haptic shared-control methods for robotic cutting under nonholonomic constraints. In 2019 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 8151–8157).

  • Rahal, R., Matarese, G., Gabiccini, M., Artoni, A., Prattichizzo, D., Robuffo Giordano, P., & Pacchierotti, C. (2020). Caring about the human operator: Haptic shared control for enhanced user comfort in robotic telemanipulation. IEEE Transactions on Haptics, 13(1), 197–203.

    Article  Google Scholar 

  • Ratliff, N., Zucker, M., Bagnell, JA., Srinivasa, S. (2009). Chomp: Gradient optimization techniques for efficient motion planning. In 2009 international conference on robotics and automation (ICRA) (pp. 489–494). IEEE.

  • Rozo, L., Jiménez, P., & Torras, C. (2013). A robot learning from demonstration framework to perform force-based manipulation tasks. Intelligent Service Robotics, 6(1), 33–51.

    Article  Google Scholar 

  • Schulman, J., Duan, Y., Ho, J., Lee, A., Awwal, I., Bradlow, H., Pan, J., Patil, S., Goldberg, K., & Abbeel, P. (2014). Motion planning with sequential convex optimization and convex collision checking. The International Journal of Robotics Research, 33(9), 1251–1270.

    Article  Google Scholar 

  • Selvaggio, M., Ghalamzan Esfahani, A., Moccia, R., Ficuciello, F., & Siciliano, B. (2019a). Haptic-guided shared control for needle grasping optimization in minimally invasive robotic surgery. In 2019 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 3617–3623). IEEE.

  • Selvaggio,M., Giordano, P.Robuffo., Ficuciello, F., & Siciliano, B. (2019b). Passive task-prioritized shared-control teleoperation with haptic guidance. In 2019 international conference on robotics and automation (ICRA) (pp. 430–436). IEEE.

  • Selvaggio, M., Abi-Farraj, F., Pacchierotti, C., Robuffo Giordano, P., & Siciliano, B. (2018). Haptic-based shared-control methods for a dual-arm system. IEEE Robotics and Automation Letters, 3(4), 4249–4256.

    Article  Google Scholar 

  • Selvaggio, M., Cacace, J., Pacchierotti, C., Ruggiero, F., & Giordano, P. R. (2021). A shared-control teleoperation architecture for nonprehensile object transportation. IEEE Transactions on Robotics, 38(1), 569–583.

    Article  Google Scholar 

  • Singh, J., Srinivasan, A. R., Neumann, G., & Kucukyilmaz, A. (2020). Haptic-guided teleoperation of a 7-dof collaborative robot arm with an identical twin master. IEEE Transactions on Haptics, 13(1), 246–252.

    Article  Google Scholar 

  • Talha, M., Ghalamzan E, A. M., Takahashi, C., Kuo, J., Ingamells, W., & Stolkin, R. (2016). Towards robotic decommissioning of legacy nuclear plant: Results of human-factors experiments with tele-robotic manipulation, and a discussion of challenges and approaches for decommissioning. In IEEE international symposium on safety, security, and rescue robotics (SSRR) (pp .166–173). IEEE.

  • Tobin, J., Biewald, L., Duan, R., Andrychowicz, M., Handa, A., Kumar, V., McGrew, B., Ray, A., Schneider, J., Welinder, P., et al. (2018). Domain randomization and generative models for robotic grasping. In 2018 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 3482–3489). IEEE

  • Vahrenkamp, N., Asfour, T., Metta, G., Sandini, G., & Dillmann, R. (2012). Manipulability analysis. In 2012 IEEE-RAS international conference on humanoid robots (Humanoids) (pp. 568–573). IEEE.

  • Vahrenkamp, N., Przybylski, M., Asfour, T. & Dillmann, R. (2011). Bimanual grasp planning. In 2011 IEEE-RAS international conference on humanoid robots (Humanoids) (pp. 493–499). IEEE

  • van Oosterhout, J., Wildenbeest, J. G., Boessenkool, H., Heemskerk, C. J., de Baar, M. R., van der Helm, F. C., & Abbink, D. A. (2015). Haptic shared control in tele-manipulation: Effects of inaccuracies in guidance on task execution. IEEE Transactions on Haptics, 8(2), 164–175.

    Article  Google Scholar 

  • Varley, J., DeChant, C., Richardson, A., Ruales, J. & Allen, P. (2017). Shape completion enabled robotic grasping. In 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 2442–2447). IEEE.

  • Xi, B., Wang, S., Ye, X., Cai, Y., Lu, T., & Wang, R. (2019). A robotic shared control teleoperation method based on learning from demonstrations. International Journal of Advanced Robotic Systems, 16(4), 1729881419857428.

    Article  Google Scholar 

  • Yang, D., & Liu, H. (2021). Human-machine shared control: New avenue to dexterous prosthetic hand manipulation. Science China Technological Sciences, 64(4), 767–773.

    Article  Google Scholar 

  • Zhou, Y., Wang, W., Guan, W., Wu, Y., Lai, H., Lu, T., & Cai, M. (2017). Visual robotic object grasping through combining RGB-d data and 3d meshes. In International conference on multimedia modeling (pp. 404–415). Springer

  • Zhu, Y., Yang, C., Wei, Q., Wu, X. & Yang, W. (2020). Human–robot shared control for humanoid manipulator trajectory planning. Industrial Robot: The International Journal of Robotics Research and Application.

  • Zucker, M., Ratliff, N., Dragan, A. D., Pivtoraiko, M., Klingensmith, M., Dellin, C. M., Bagnell, J. A., & Srinivasa, S. S. (2013). Chomp: Covariant hamiltonian optimization for motion planning. The International Journal of Robotics Research, 32(9–10), 1164–1193.

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Rahaf Rahal.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This project was partially funded by EU H2020 RoMaNS, 645582, and EPSRC EP/M026477/1, National Centre for Nuclear Robotic.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file 1 (mp4 101424 KB)

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rahal, R., Ghalamzan-E., A.M., Abi-Farraj, F. et al. Haptic-guided grasping to minimise torque effort during robotic telemanipulation. Auton Robot 47, 405–423 (2023). https://doi.org/10.1007/s10514-023-10096-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10514-023-10096-7

Keywords

Navigation