Skip to main content
SpringerLink
Log in

Human-machine shared control: New avenue to dexterous prosthetic hand manipulation

  • Article
  • Published:
Science China Technological Sciences Aims and scope Submit manuscript

Abstract

At present, albeit the dexterous hand prostheses of multiple degrees of freedom (DOFs) have become prosperous on the market, the user’s demand on intuitively operating these devices have not been well addressed so that their acceptance rate is relatively low. The unintuitive control method and inadequate sensory feedback are frequently cited as the two barriers to the successful application of these dexterous products. Recently, driven by the wave of artificial intelligence (AI), a series of shared control methods have emerged, in which “bodily function” (myoelectric control) and “artificial intelligence” (local autonomy, computer vision, etc.) are tightly integrated, and provided a new conceptual solution for the intuitive operation of dexterous prostheses. In this paper, the background and development trends of this type of methods are described in detail, and the potential development directions and the key technologies that need breakthroughs are indicated. In practice, we instantiate this shared control strategy by proposing a new method combining simultaneous myoelectric control, multi-finger grasp autonomy, and augmented reality (AR) feedback together. This method “divides” the human sophisticated reach-and-grasp task into several subtasks, and then “conquers” them by using different strategies from either human or machine perspective. It is highly expected that the shared control methods with hybrid human-machine intelligence could address the control problem of dexterous prostheses.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Atkins D J, Heard D C Y, Donovan W H. Epidemiologic overview of individuals with upper-limb loss and their reported research priorities. J Prosthet Orthot, 1996, 8: 2–11

    Article  Google Scholar 

  2. Belter J T, Segil J L, Dollar A M, et al. Mechanical design and performance specifications of anthropomorphic prosthetic hands: A review. J Rehabil Res Dev, 2013, 50: 599–618

    Article  Google Scholar 

  3. Micera S, Carpaneto J, Raspopovic S. Control of hand prostheses using peripheral information. IEEE Rev Biomed Eng, 2010, 3: 48–68

    Article  Google Scholar 

  4. Fougner A, Stavdahl O, Kyberd P J, et al. Control of upper limb prostheses: Terminology and proportional myoelectric control: A review. IEEE Trans Neural Syst Rehab Eng, 2012, 20: 663–677

    Article  Google Scholar 

  5. Kuiken T A, Li G, Lock B A, et al. Targeted muscle reinnervation for real-time myoelectric control of multifunction artificial arms. JAMA, 2009, 301: 619–628

    Article  Google Scholar 

  6. Farina D, Vujaklija I, Sartori M, et al. Man/machine interface based on the discharge timings of spinal motor neurons after targeted muscle reinnervation. Nat Biomed Eng, 2017, 1: 0025

    Article  Google Scholar 

  7. Collinger J L, Wodlinger B, Downey J E, et al. High-performance neuroprosthetic control by an individual with tetraplegia. Lancet, 2013, 381: 557–564

    Article  Google Scholar 

  8. Osborn L E, Dragomir A, Betthauser J L, et al. Prosthesis with neuromorphic multilayered e-dermis perceives touch and pain. Sci Robot, 2018, 3: eaat3818

    Article  Google Scholar 

  9. Fang Y, Hettiarachchi N, Zhou D, et al. Multi-modal sensing techniques for interfacing hand prostheses: A review. IEEE Sens J, 2015, 15: 6065–6076

    Article  Google Scholar 

  10. Farina D, Ning Jiang D, Rehbaum H, et al. The extraction of neural information from the surface EMG for the control of upper-limb prostheses: Emerging avenues and challenges. IEEE Trans Neural Syst Rehab Eng, 2014, 22: 797–809

    Article  Google Scholar 

  11. Englehart K, Hudgins B. A robust, real-time control scheme for multifunction myoelectric control. IEEE Trans Biomed Eng, 2003, 50: 848–854

    Article  Google Scholar 

  12. Ning J, Dosen S, Muller K R, et al. Myoelectric control of artificial limbs: Is there a need to change focus? IEEE Signal Process Mag, 2012, 29: 152–150

    Article  Google Scholar 

  13. Castellini C, Artemiadis P, Wininger M, et al. Proceedings of the first workshop on Peripheral Machine Interfaces: Going beyond traditional surface electromyography. Front Neurorobot, 2014, 8: 22

    Article  Google Scholar 

  14. Atzori M, Gijsberts A, Castellini C, et al. Electromyography data for non-invasive naturally-controlled robotic hand prostheses. Sci Data, 2014, 1: 140053

    Article  Google Scholar 

  15. Geng W, Du Y, Jin W, et al. Gesture recognition by instantaneous surface EMG images. Sci Rep, 2016, 6: 36571

    Article  Google Scholar 

  16. Zia ur Rehman M, Gilani S, Waris A, et al. Stacked sparse autoencoders for EMG-based classification of hand motions: A comparative multi day analyses between surface and intramuscular EMG. Appl Sci, 2018, 8: 1126

    Article  Google Scholar 

  17. Olsson A E, Sager P, Andersson E, et al. Extraction of multi-labelled movement information from the raw HD-sEMG image with time-domain depth. Sci Rep, 2019, 9: 7244

    Article  Google Scholar 

  18. Phinyomark A, Scheme E. EMG pattern recognition in the era of big data and deep learning. Big Data Cognitive Comput, 2018, 2: 21

    Article  Google Scholar 

  19. Velliste M, Perel S, Spalding M C, et al. Cortical control of a prosthetic arm for self-feeding. Nature, 2008, 453: 1098–1101

    Article  Google Scholar 

  20. Hochberg L R, Bacher D, Jarosiewicz B, et al. Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature, 2012, 485: 372–375

    Article  Google Scholar 

  21. Ahmadizadeh C, Merhi L K, Pousett B, et al. Toward intuitive prosthetic control: Solving common issues using force myography, surface electromyography, and pattern recognition in a pilot case study. IEEE Robot Automat Mag, 2017, 24: 102–111

    Article  Google Scholar 

  22. Fazli S, Mehnert J, Steinbrink J, et al. Enhanced performance by a hybrid NIRS-EEG brain computer interface. Neuroimage, 2012, 59: 519–529

    Article  Google Scholar 

  23. Bohg J, Morales A, Asfour T, et al. Data-driven grasp synthesis-A survey. IEEE Trans Robot, 2014, 30: 289–309

    Article  Google Scholar 

  24. Lenz I, Lee H, Saxena A. Deep learning for detecting robotic grasps. Int J Robot Res, 2015, 34: 705–724

    Article  Google Scholar 

  25. Levine S, Pastor P, Krizhevsky A, et al. Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection. Int J Robot Res, 2018, 37: 421–436

    Article  Google Scholar 

  26. Cipriani C, Antfolk C, Balkenius C, et al. A novel concept for a prosthetic hand with a bidirectional interface: A feasibility study. IEEE Trans Biomed Eng, 2009, 56: 2739–2743

    Article  Google Scholar 

  27. Stephens-Fripp B, Alici G, Mutlu R. A review ofnon-invasive sensory feedback methods for transradial prosthetic hands. IEEE Access, 2018, 6: 6878–6899

    Article  Google Scholar 

  28. Markovic M, Dosen S, Cipriani C, et al. Stereovision and augmented reality for closed-loop control ofgrasping in hand prostheses. J Neural Eng, 2014, 11: 046001

    Article  Google Scholar 

  29. Yang D, Gu Y, Thakor N V, et al. Improving the functionality, robustness, and adaptability of myoelectric control for dexterous motion restoration. Exp Brain Res, 2019, 237: 291–311

    Article  Google Scholar 

  30. Hahne J M, Schweisfurth M A, Koppe M, et al. Simultaneous control of multiple functions of bionic hand prostheses: Performance and robustness in end users. Sci Robot, 2018, 3: eaat3630

    Article  Google Scholar 

  31. Cipriani C, Zaccone F, Micera S, et al. On the shared control of an EMG-controlled prosthetic hand: Analysis of user-prosthesis interaction. IEEE Trans Robot, 2008, 24: 170–184

    Article  Google Scholar 

  32. Quigley M, Salisbury C, Ng A Y, et al. Mechatronic design of an integrated robotic hand. Int J Robot Res, 2014, 33: 706–720

    Article  Google Scholar 

  33. Zhuang K Z, Sommer N, Mendez V, et al. Shared human-robot proportional control of a dexterous myoelectric prosthesis. Nat Mach Intell, 2019, 1: 400–411

    Article  Google Scholar 

  34. Ghazaei G, Alameer A, Degenaar P, et al. Deep learning-based artificial vision for grasp classification in myoelectric hands. J Neural Eng, 2017, 14: 036025

    Article  Google Scholar 

  35. Markovic M, Karnal H, Graimann B, et al. GLIMPSE: Google Glass interface for sensory feedback in myoelectric hand prostheses. J Neural Eng, 2017, 14: 036007

    Article  Google Scholar 

  36. Bullock I M, Feix T, Dollar A M. The Yale human grasping dataset: Grasp, object, and task data in household and machine shop environments. Int J Robot Res, 2015, 34: 251–255

    Article  Google Scholar 

  37. Romero J, Feix T, Ek C H, et al. Extracting postural synergies for robotic grasping. IEEE Trans Robot, 2013, 29: 1342–1352

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Harbin Institute of Technology, State Key Laboratory of Robotics and System, Harbin, 150080, China

    DaPeng Yang & Hong Liu

  2. Harbin Institute of Technology, Artificial Intelligence Laboratory, Harbin, 150080, China

    DaPeng Yang

Authors
  1. DaPeng Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Hong Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to DaPeng Yang.

Additional information

This work was supported by the National Key R&D Program of China (Grant No. 2018YFB1307201), the National Natural Science Foundation of China (Grant No. 51675123), and the Postdoctoral Scientific Research Development Fund (Grant No. LBH-W18058).

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yang, D., Liu, H. Human-machine shared control: New avenue to dexterous prosthetic hand manipulation. Sci. China Technol. Sci. 64, 767–773 (2021). https://doi.org/10.1007/s11431-020-1710-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11431-020-1710-y

Keywords

Search

Navigation