Sub-optimally solving actuator redundancy in a hybrid neuroprosthetic system with a multi-layer neural network structure

  • Xuefeng Bao
  • Zhi-Hong Mao
  • Paul Munro
  • Ziyue Sun
  • Nitin SharmaEmail author
Regular Paper


Functional electrical stimulation (FES) has recently been proposed as a supplementary torque assist in lower-limb powered exoskeletons for persons with paraplegia. In the combined system, also known as a hybrid neuroprosthesis, both FES-assist and the exoskeleton act to generate lower-limb torques to achieve standing and walking functions. Due to this actuator redundancy, we are motivated to optimally allocate FES-assist and exoskeleton torque based on a performance index that penalizes FES overuse to minimize muscle fatigue while also minimizing regulation or tracking errors. Traditional optimal control approaches need a system model to optimize; however, it is often difficult to formulate a musculoskeletal model that accurately predicts muscle responses due to FES. In this paper, we use a novel identification and control structure that contains a recurrent neural network (RNN) and several feedforward neural networks (FNNs). The RNN is trained by supervised learning to identify the system dynamics, while the FNNs are trained by a reinforcement learning method to provide sub-optimal control actions. The output layer of each FNN has its unique activation functions, so that the asymmetric constraint of FES and the symmetric constraint of exoskeleton motor control input can be realized. This new structure is experimentally validated on a seated human participant using a single joint hybrid neuroprosthesis.


Hybrid neuroprosthesis Actuator redundancy Rehabilitation Neural network Reinforcement learning 



  1. Akpan, V., Hassapis, G.: Nonlinear model identification and adaptive model predictive control using neural networks. ISA Trans. 50(2), 177–194 (2011)CrossRefGoogle Scholar
  2. Alibeji, N.A., Molazadeh, V., Dicianno, B.E., Sharma, N.: A control scheme that uses dynamic postural synergies to coordinate a hybrid walking neuroprosthesis: theory and experiments. Front. Neurosci. 12, 159 (2018). (Online)
  3. Alibeji, N.A., Molazadeh, V., Moore-Cligenpeel, F., Sharma, N.: A muscle synergy inspired control design to coordinate functional electrical stimulation and a powered exoskeleton. IEEE Control Syst. Mag. 38, 35–60 (2018) (conditionally accepted) Google Scholar
  4. Alibeji, N., Kirsch, N., Sharma, N.: A muscle synergy-inspired adaptive control scheme for a hybrid walking neuroprosthesis. Front. Bioeng. Biotechnol. 3, 203 (2015)CrossRefGoogle Scholar
  5. Alibeji, N., Kirsch, N., Sharma, N.: An adaptive low-dimensional control to compensate for actuator redundancy and fes-induced muscle fatigue in a hybrid neuroprosthesis. Control Eng. Pract. 59, 204–219 (2017)CrossRefGoogle Scholar
  6. Anaya, F., Thangavel, P., Yu, H.: Hybrid fes-robotic gait rehabilitation technologies: a review on mechanical design, actuation, and control strategies. Int. J. Intell. Robot. Appl. 2(1), 1–28 (2018)CrossRefGoogle Scholar
  7. Bao, X., Dicianno, B., Sharma, N.: Model predictive control of a feedback linearized hybrid neuroprosthetic system with a barrier penalty. J. Comput. Nonlinear Dyn. (2019) (in press)Google Scholar
  8. Bao, X., Sun, Z., Sharma, N.: A recurrent neural network based mpc for a hybrid neuroprosthesis system. In: 2017 IEEE 56th Annual Conference on Decision and Control (CDC), IEEE, Melbourne, 12–15 Dec 2017Google Scholar
  9. Beaufays, F., Wan, E.: Relating real-time backpropagation and backpropagation-through-time: an application of flow graph interreciprocity. Neural Comput. 6(2), 296–306 (1994)CrossRefGoogle Scholar
  10. Bickel, C., Gregory, C., Dean, J.: Motor unit recruitment during neuromuscular electrical stimulation: a critical appraisal. Eur. J. Appl. Physiol. 111(10), 2399–2407 (2011)CrossRefGoogle Scholar
  11. Chen, Y.Q., Yin, T., Babri, H.A.: A stochastic backpropagation algorithm for training neural networks. In: Proceedings of ICICS, 1997 International Conference on Information, Communications and Signal Processing. Theme: Trends in Information Systems Engineering and Wireless Multimedia Communications (Cat No.97TH8237), vol. 2, pp. 703-707. IEEE, Singapore, 12 Sept 1997Google Scholar
  12. del Ama, A., Gil-Agudo, Á., Pons, J., Moreno, J.: Hybrid FES-robot cooperative control of ambulatory gait rehabilitation exoskeleton. J. Neuroeng. Rehabil. 11(1), 27 (2014)CrossRefGoogle Scholar
  13. Dodson, A.: A novel user-controlled assisted standing control system for a hybrid neuroprosthesis, Master’s Thesis, University of Pittsburgh (2018)Google Scholar
  14. Durfee, W.K.: Gait restoration by functional electrical stimulation. Climbing and Walking Robots, pp. 19–26. Springer, Berlin, Heidelberg (2006)CrossRefGoogle Scholar
  15. Durfee, W.K., Hausdorff, J.M.: Regulating knee joint position by combining electrical stimulation with a controllable friction brake. Ann. Biomed. Eng. 18(6), 575–596 (1990)CrossRefGoogle Scholar
  16. Goldfarb, M., Korkowski, K., Harrold, B., Durfee, W.: Preliminary evaluation of a controlled-brake orthosis for FES-aided gait. IEEE Trans. Neural Syst. Rehabil. Eng. 11(3), 241–248 (2003)CrossRefGoogle Scholar
  17. Goodfellow, I., Bengio, Y., Courville, A., Bengio, Y.: Deep Learning, vol. 1. MIT press, Cambridge (2016)zbMATHGoogle Scholar
  18. Graichen, K., Käpernick, B.: A real-time gradient method for nonlinear model predictive control. INTECH Open Access Publisher, London (2012)CrossRefzbMATHGoogle Scholar
  19. Grondman, I., Busoniu, L., Lopes, G., Babuska, R.: A survey of actor-critic reinforcement learning: standard and natural policy gradients. IEEE Trans. Syst. Man Cybern. C Appl. Rev. 42(6), 1291–1307 (2012)CrossRefGoogle Scholar
  20. Ha, K.H., Murray, S.A., Goldfarb, M.: An approach for the cooperative control of FES with a powered exoskeleton during level walking for persons with paraplegia. IEEE Trans. Neural Syst. Rehabil. Eng. 24(4), 455–466 (2015)CrossRefGoogle Scholar
  21. Jagodnik, K.M., Thomas, P.S., Van Den Bogert, A.J., Branicky, M.S., Kirsch, R.F.: Human-like rewards to train a reinforcement learning controller for planar arm movement. IEEE Trans. Human Mach. Syst. 46(5), 723–733 (2016)CrossRefGoogle Scholar
  22. Jagodnik, K.M., Thomas, P.S., van den Bogert, A.J., Branicky, M.S., Kirsch, R.F.: Training an actor-critic reinforcement learning controller for arm movement using human-generated rewards. IEEE Trans. Neural Syst. Rehabil. Eng. 25(10), 1892–1905 (2017)CrossRefGoogle Scholar
  23. Jordan, M., Rumelhart, D.: Forward models: supervised learning with a distal teacher. Cogn. Sci. 16(3), 307–354 (1992)CrossRefGoogle Scholar
  24. Kayacan, E., Kayacan, E., Ramon, H., Saeys, W.: Robust tube-based decentralized nonlinear model predictive control of an autonomous tractor-trailer system. IEEE/ASME Trans. Mechatron. 20(1), 447–456 (2015)CrossRefGoogle Scholar
  25. Kirsch, N., Alibeji, N., Fisher, L., Gregory, C., Sharma, N.: A semi-active hybrid neuroprosthesis for restoring lower limb function in paraplegics. Conf Proc IEEE Eng Med Biol Soc. 2014, 2557–2560 (2014). Google Scholar
  26. Kirsch, N., Alibeji, N., Sharma, N.: Nonlinear model predictive control of functional electrical stimulation. Control Eng. Pract. 58, 319–331 (2017)CrossRefGoogle Scholar
  27. Kirsch, N., Bao, X., Alibeji, N., Dicianno, B., Sharma, N.: Model-based dynamic control allocation in a hybrid neuroprosthesis. IEEE Trans. Neural Syst. Rehabil. Eng. 26(1), 224–232 (2018)CrossRefGoogle Scholar
  28. Kobetic, R., Marsolais, B.: Synthesis of paraplegic gait with multichannel functional neuromuscular stimulation. IEEE Trans. Rehabil. Eng. 2(2), 66–79 (1994)CrossRefGoogle Scholar
  29. Kobetic, R., To, C., Schnellenberger, J., Audu, M., Bulea, T., Gaudio, R., Pinault, G., Tashman, S., Triolo, R.: Development of hybrid orthosis for standing, walking, and stair climbing after spinal cord injury. J. Rehabil. Res. Dev. 46(3), 447–462 (2009)CrossRefGoogle Scholar
  30. Lewis, F., Vrabie, D.: Reinforcement learning and adaptive dynamic programming for feedback control. IEEE Circuits Syst. Mag. 9(3), 32–50 (2009)CrossRefGoogle Scholar
  31. Lin, L.J., 1993. Reinforcement learning for robots using neural networks. Ph.D. Thesis, Carnegie Mellon University, Pittsburgh PA. Technical Report CMU-CS-93-103 (1993)Google Scholar
  32. Mayne, D., Kerrigan, E., van Wyk, E., Falugi, P.: Tube-based robust nonlinear model predictive control. Int. J. Robust Nonlinear Control 21(11), 1341–1353 (2011)MathSciNetCrossRefzbMATHGoogle Scholar
  33. McCallum, R.A.: Hidden state and reinforcement learning with instance-based state identification. IEEE Trans. Syst. Man Cybern. Part B 26(3), 464–473 (1996)CrossRefGoogle Scholar
  34. Munro, P.: A dual back-propagation scheme for scalar reward learning. In: Ninth Annual Conference of the Cognitive Science Society, pp. 165–176 (1987)Google Scholar
  35. Peters, J., Schaal, S.: Policy gradient methods for robotics. In 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2219-2225. IEEE, Beijing, 9–15 Oct 2006Google Scholar
  36. Popović, D., Stein, R., Oğuztöreli, M., Lebiedowska, M., Jonić, S.: Optimal control of walking with functional electrical stimulation: a computer simulation study. IEEE Trans. Rehabil. Eng. 7(1), 69–79 (1999)CrossRefGoogle Scholar
  37. Riener, R., Quintern, J., Schmidt, G.: Biomechanical model of the human knee evaluated by neuromuscular stimulation. J. Biomech. 29, 1157–1167 (1996)CrossRefGoogle Scholar
  38. Schaefer, A.M., Schneegass, D., Sterzing, V., Udluft, S.: A neural reinforcement learning approach to gas turbine control. In: 2007 International Joint Conference on Neural Networks, pp. 1691–1696. IEEE (2007)Google Scholar
  39. Schäfer, A.M., Udluft, S., et al.: Solving partially observable reinforcement learning problems with recurrent neural networks. In: Workshop Proceedings of the European Conference on Machine Learning, pp. 71–81 (2005)Google Scholar
  40. Schäfer, A.M., Udluft, S., Zimmermann, H.G.: A recurrent control neural network for data efficient reinforcement learning. In: 2007 IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning, pp. 151–157. IEEE (2007)Google Scholar
  41. Sharma, N., Kirsch, N.A., Alibeji, N.A., Dixon, W.E.: A non-linear control method to compensate for muscle fatigue during neuromuscular electrical stimulation. Front. Robot. AI 4, 68 (2017). (Online)
  42. Sharma, N., Stegath, K., Gregory, C.M., Dixon, W.E.: Nonlinear neuromuscular electrical stimulation tracking control of a human limb. IEEE Trans. Neural Syst. Rehabil. Eng. 17(6), 576–584 (2012)CrossRefGoogle Scholar
  43. Sharma, N., Mushahwar, V., Stein, R.: Dynamic optimization of FES and orthosis-based walking using simple models. IEEE Trans. Neural Syst. Rehabil. Eng. 22, 114–126 (2014)CrossRefGoogle Scholar
  44. Silver, D., Huang, A., Maddison, C., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al.: Mastering the game of go with deep neural networks and tree search. Nature 529(7587), 484–489 (2016)CrossRefGoogle Scholar
  45. Sutton, R., McAllester, D., Singh, S., Mansour, Y., et al.: Policy gradient methods for reinforcement learning with function approximation. NIPS 99, 1057–1063 (1999)Google Scholar
  46. Vallery, H., Stützle, T., Buss, M., Abel, D.: Control of a hybrid motor prosthesis for the knee joint. IFAC Proc. Vol. 38(1), 76–81 (2005)CrossRefGoogle Scholar
  47. Vrabie, D., Vamvoudakis, K.G., Lewis, F.L.: Optimal adaptive control and differential games by reinforcement learning principles, vol. 2. IET Press (2013)Google Scholar
  48. Wang, F.-Y., Zhang, H., Liu, D.: Adaptive dynamic programming: an introduction. IEEE Comput. Intell. Mag. 4(2), 39–47 (2009)CrossRefGoogle Scholar
  49. Werbos, P.J.: Backpropagation through time: what it does and how to do it. Proc. IEEE 78(10), 1550–1560 (1990)CrossRefGoogle Scholar
  50. Williams, R.: Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn. 8(3–4), 229–256 (1992)zbMATHGoogle Scholar
  51. Witten, I.H., Frank, E., Hall, M.A., Pal, C.J.: Data Mining: Practical Machine Learning Tools and Techniques. Morgan Kaufmann, Burlington (2016)Google Scholar

Copyright information

© Springer Nature Singapore Pte Ltd. 2019

Authors and Affiliations

  1. 1.Department of Mechanical Engineering and Materials ScienceUniversity of PittsburghPittsburghUSA
  2. 2.Department of Electrical and Computer EngineeringUniversity of PittsburghPittsburghUSA
  3. 3.Department of BioengineeringUniversity of PittsburghPittsburghUSA

Personalised recommendations