Skip to main content
SpringerLink
Log in

Sub-optimally solving actuator redundancy in a hybrid neuroprosthetic system with a multi-layer neural network structure

  • Regular Paper
  • Published:
International Journal of Intelligent Robotics and Applications Aims and scope Submit manuscript

Abstract

Functional electrical stimulation (FES) has recently been proposed as a supplementary torque assist in lower-limb powered exoskeletons for persons with paraplegia. In the combined system, also known as a hybrid neuroprosthesis, both FES-assist and the exoskeleton act to generate lower-limb torques to achieve standing and walking functions. Due to this actuator redundancy, we are motivated to optimally allocate FES-assist and exoskeleton torque based on a performance index that penalizes FES overuse to minimize muscle fatigue while also minimizing regulation or tracking errors. Traditional optimal control approaches need a system model to optimize; however, it is often difficult to formulate a musculoskeletal model that accurately predicts muscle responses due to FES. In this paper, we use a novel identification and control structure that contains a recurrent neural network (RNN) and several feedforward neural networks (FNNs). The RNN is trained by supervised learning to identify the system dynamics, while the FNNs are trained by a reinforcement learning method to provide sub-optimal control actions. The output layer of each FNN has its unique activation functions, so that the asymmetric constraint of FES and the symmetric constraint of exoskeleton motor control input can be realized. This new structure is experimentally validated on a seated human participant using a single joint hybrid neuroprosthesis.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  • Akpan, V., Hassapis, G.: Nonlinear model identification and adaptive model predictive control using neural networks. ISA Trans. 50(2), 177–194 (2011)

    Article  Google Scholar 

  • Alibeji, N.A., Molazadeh, V., Dicianno, B.E., Sharma, N.: A control scheme that uses dynamic postural synergies to coordinate a hybrid walking neuroprosthesis: theory and experiments. Front. Neurosci. 12, 159 (2018). https://www.frontiersin.org/article/10.3389/fnins.2018.00159 (Online)

  • Alibeji, N.A., Molazadeh, V., Moore-Cligenpeel, F., Sharma, N.: A muscle synergy inspired control design to coordinate functional electrical stimulation and a powered exoskeleton. IEEE Control Syst. Mag. 38, 35–60 (2018) (conditionally accepted)

  • Alibeji, N., Kirsch, N., Sharma, N.: A muscle synergy-inspired adaptive control scheme for a hybrid walking neuroprosthesis. Front. Bioeng. Biotechnol. 3, 203 (2015)

    Article  Google Scholar 

  • Alibeji, N., Kirsch, N., Sharma, N.: An adaptive low-dimensional control to compensate for actuator redundancy and fes-induced muscle fatigue in a hybrid neuroprosthesis. Control Eng. Pract. 59, 204–219 (2017)

    Article  Google Scholar 

  • Anaya, F., Thangavel, P., Yu, H.: Hybrid fes-robotic gait rehabilitation technologies: a review on mechanical design, actuation, and control strategies. Int. J. Intell. Robot. Appl. 2(1), 1–28 (2018)

    Article  Google Scholar 

  • Bao, X., Dicianno, B., Sharma, N.: Model predictive control of a feedback linearized hybrid neuroprosthetic system with a barrier penalty. J. Comput. Nonlinear Dyn. (2019) (in press)

  • Bao, X., Sun, Z., Sharma, N.: A recurrent neural network based mpc for a hybrid neuroprosthesis system. In: 2017 IEEE 56th Annual Conference on Decision and Control (CDC), IEEE, Melbourne, 12–15 Dec 2017

  • Beaufays, F., Wan, E.: Relating real-time backpropagation and backpropagation-through-time: an application of flow graph interreciprocity. Neural Comput. 6(2), 296–306 (1994)

    Article  Google Scholar 

  • Bickel, C., Gregory, C., Dean, J.: Motor unit recruitment during neuromuscular electrical stimulation: a critical appraisal. Eur. J. Appl. Physiol. 111(10), 2399–2407 (2011)

    Article  Google Scholar 

  • Chen, Y.Q., Yin, T., Babri, H.A.: A stochastic backpropagation algorithm for training neural networks. In: Proceedings of ICICS, 1997 International Conference on Information, Communications and Signal Processing. Theme: Trends in Information Systems Engineering and Wireless Multimedia Communications (Cat No.97TH8237), vol. 2, pp. 703-707. IEEE, Singapore, 12 Sept 1997

  • del Ama, A., Gil-Agudo, Á., Pons, J., Moreno, J.: Hybrid FES-robot cooperative control of ambulatory gait rehabilitation exoskeleton. J. Neuroeng. Rehabil. 11(1), 27 (2014)

    Article  Google Scholar 

  • Dodson, A.: A novel user-controlled assisted standing control system for a hybrid neuroprosthesis, Master’s Thesis, University of Pittsburgh (2018)

  • Durfee, W.K.: Gait restoration by functional electrical stimulation. Climbing and Walking Robots, pp. 19–26. Springer, Berlin, Heidelberg (2006)

    Chapter  Google Scholar 

  • Durfee, W.K., Hausdorff, J.M.: Regulating knee joint position by combining electrical stimulation with a controllable friction brake. Ann. Biomed. Eng. 18(6), 575–596 (1990)

    Article  Google Scholar 

  • Goldfarb, M., Korkowski, K., Harrold, B., Durfee, W.: Preliminary evaluation of a controlled-brake orthosis for FES-aided gait. IEEE Trans. Neural Syst. Rehabil. Eng. 11(3), 241–248 (2003)

    Article  Google Scholar 

  • Goodfellow, I., Bengio, Y., Courville, A., Bengio, Y.: Deep Learning, vol. 1. MIT press, Cambridge (2016)

    MATH  Google Scholar 

  • Graichen, K., Käpernick, B.: A real-time gradient method for nonlinear model predictive control. INTECH Open Access Publisher, London (2012)

    Book  MATH  Google Scholar 

  • Grondman, I., Busoniu, L., Lopes, G., Babuska, R.: A survey of actor-critic reinforcement learning: standard and natural policy gradients. IEEE Trans. Syst. Man Cybern. C Appl. Rev. 42(6), 1291–1307 (2012)

    Article  Google Scholar 

  • Ha, K.H., Murray, S.A., Goldfarb, M.: An approach for the cooperative control of FES with a powered exoskeleton during level walking for persons with paraplegia. IEEE Trans. Neural Syst. Rehabil. Eng. 24(4), 455–466 (2015)

    Article  Google Scholar 

  • Jagodnik, K.M., Thomas, P.S., Van Den Bogert, A.J., Branicky, M.S., Kirsch, R.F.: Human-like rewards to train a reinforcement learning controller for planar arm movement. IEEE Trans. Human Mach. Syst. 46(5), 723–733 (2016)

    Article  Google Scholar 

  • Jagodnik, K.M., Thomas, P.S., van den Bogert, A.J., Branicky, M.S., Kirsch, R.F.: Training an actor-critic reinforcement learning controller for arm movement using human-generated rewards. IEEE Trans. Neural Syst. Rehabil. Eng. 25(10), 1892–1905 (2017)

    Article  Google Scholar 

  • Jordan, M., Rumelhart, D.: Forward models: supervised learning with a distal teacher. Cogn. Sci. 16(3), 307–354 (1992)

    Article  Google Scholar 

  • Kayacan, E., Kayacan, E., Ramon, H., Saeys, W.: Robust tube-based decentralized nonlinear model predictive control of an autonomous tractor-trailer system. IEEE/ASME Trans. Mechatron. 20(1), 447–456 (2015)

    Article  Google Scholar 

  • Kirsch, N., Alibeji, N., Fisher, L., Gregory, C., Sharma, N.: A semi-active hybrid neuroprosthesis for restoring lower limb function in paraplegics. Conf Proc IEEE Eng Med Biol Soc. 2014, 2557–2560 (2014). https://doi.org/10.1109/EMBC.2014.6944144

    Google Scholar 

  • Kirsch, N., Alibeji, N., Sharma, N.: Nonlinear model predictive control of functional electrical stimulation. Control Eng. Pract. 58, 319–331 (2017)

    Article  Google Scholar 

  • Kirsch, N., Bao, X., Alibeji, N., Dicianno, B., Sharma, N.: Model-based dynamic control allocation in a hybrid neuroprosthesis. IEEE Trans. Neural Syst. Rehabil. Eng. 26(1), 224–232 (2018)

    Article  Google Scholar 

  • Kobetic, R., Marsolais, B.: Synthesis of paraplegic gait with multichannel functional neuromuscular stimulation. IEEE Trans. Rehabil. Eng. 2(2), 66–79 (1994)

    Article  Google Scholar 

  • Kobetic, R., To, C., Schnellenberger, J., Audu, M., Bulea, T., Gaudio, R., Pinault, G., Tashman, S., Triolo, R.: Development of hybrid orthosis for standing, walking, and stair climbing after spinal cord injury. J. Rehabil. Res. Dev. 46(3), 447–462 (2009)

    Article  Google Scholar 

  • Lewis, F., Vrabie, D.: Reinforcement learning and adaptive dynamic programming for feedback control. IEEE Circuits Syst. Mag. 9(3), 32–50 (2009)

    Article  Google Scholar 

  • Lin, L.J., 1993. Reinforcement learning for robots using neural networks. Ph.D. Thesis, Carnegie Mellon University, Pittsburgh PA. Technical Report CMU-CS-93-103 (1993)

  • Mayne, D., Kerrigan, E., van Wyk, E., Falugi, P.: Tube-based robust nonlinear model predictive control. Int. J. Robust Nonlinear Control 21(11), 1341–1353 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  • McCallum, R.A.: Hidden state and reinforcement learning with instance-based state identification. IEEE Trans. Syst. Man Cybern. Part B 26(3), 464–473 (1996)

    Article  Google Scholar 

  • Munro, P.: A dual back-propagation scheme for scalar reward learning. In: Ninth Annual Conference of the Cognitive Science Society, pp. 165–176 (1987)

  • Peters, J., Schaal, S.: Policy gradient methods for robotics. In 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2219-2225. IEEE, Beijing, 9–15 Oct 2006

  • Popović, D., Stein, R., Oğuztöreli, M., Lebiedowska, M., Jonić, S.: Optimal control of walking with functional electrical stimulation: a computer simulation study. IEEE Trans. Rehabil. Eng. 7(1), 69–79 (1999)

    Article  Google Scholar 

  • Riener, R., Quintern, J., Schmidt, G.: Biomechanical model of the human knee evaluated by neuromuscular stimulation. J. Biomech. 29, 1157–1167 (1996)

    Article  Google Scholar 

  • Schaefer, A.M., Schneegass, D., Sterzing, V., Udluft, S.: A neural reinforcement learning approach to gas turbine control. In: 2007 International Joint Conference on Neural Networks, pp. 1691–1696. IEEE (2007)

  • Schäfer, A.M., Udluft, S., et al.: Solving partially observable reinforcement learning problems with recurrent neural networks. In: Workshop Proceedings of the European Conference on Machine Learning, pp. 71–81 (2005)

  • Schäfer, A.M., Udluft, S., Zimmermann, H.G.: A recurrent control neural network for data efficient reinforcement learning. In: 2007 IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning, pp. 151–157. IEEE (2007)

  • Sharma, N., Kirsch, N.A., Alibeji, N.A., Dixon, W.E.: A non-linear control method to compensate for muscle fatigue during neuromuscular electrical stimulation. Front. Robot. AI 4, 68 (2017). https://www.frontiersin.org/article/10.3389/frobt.2017.00068 (Online)

  • Sharma, N., Stegath, K., Gregory, C.M., Dixon, W.E.: Nonlinear neuromuscular electrical stimulation tracking control of a human limb. IEEE Trans. Neural Syst. Rehabil. Eng. 17(6), 576–584 (2012)

    Article  Google Scholar 

  • Sharma, N., Mushahwar, V., Stein, R.: Dynamic optimization of FES and orthosis-based walking using simple models. IEEE Trans. Neural Syst. Rehabil. Eng. 22, 114–126 (2014)

    Article  Google Scholar 

  • Silver, D., Huang, A., Maddison, C., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al.: Mastering the game of go with deep neural networks and tree search. Nature 529(7587), 484–489 (2016)

    Article  Google Scholar 

  • Sutton, R., McAllester, D., Singh, S., Mansour, Y., et al.: Policy gradient methods for reinforcement learning with function approximation. NIPS 99, 1057–1063 (1999)

    Google Scholar 

  • Vallery, H., Stützle, T., Buss, M., Abel, D.: Control of a hybrid motor prosthesis for the knee joint. IFAC Proc. Vol. 38(1), 76–81 (2005)

    Article  Google Scholar 

  • Vrabie, D., Vamvoudakis, K.G., Lewis, F.L.: Optimal adaptive control and differential games by reinforcement learning principles, vol. 2. IET Press (2013)

  • Wang, F.-Y., Zhang, H., Liu, D.: Adaptive dynamic programming: an introduction. IEEE Comput. Intell. Mag. 4(2), 39–47 (2009)

    Article  Google Scholar 

  • Werbos, P.J.: Backpropagation through time: what it does and how to do it. Proc. IEEE 78(10), 1550–1560 (1990)

    Article  Google Scholar 

  • Williams, R.: Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn. 8(3–4), 229–256 (1992)

    MATH  Google Scholar 

  • Witten, I.H., Frank, E., Hall, M.A., Pal, C.J.: Data Mining: Practical Machine Learning Tools and Techniques. Morgan Kaufmann, Burlington (2016)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Mechanical Engineering and Materials Science, University of Pittsburgh, Pittsburgh, PA, 15261, USA

    Xuefeng Bao, Ziyue Sun & Nitin Sharma

  2. Department of Electrical and Computer Engineering, University of Pittsburgh, Pittsburgh, PA, 15261, USA

    Zhi-Hong Mao & Paul Munro

  3. Department of Bioengineering, University of Pittsburgh, Pittsburgh, PA, 15261, USA

    Zhi-Hong Mao, Paul Munro & Nitin Sharma

Authors
  1. Xuefeng Bao
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Zhi-Hong Mao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Paul Munro
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Ziyue Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Nitin Sharma
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nitin Sharma.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bao, X., Mao, ZH., Munro, P. et al. Sub-optimally solving actuator redundancy in a hybrid neuroprosthetic system with a multi-layer neural network structure. Int J Intell Robot Appl 3, 298–313 (2019). https://doi.org/10.1007/s41315-019-00100-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s41315-019-00100-8

Keywords

Search

Navigation