Abstract
Functional electrical stimulation (FES) has recently been proposed as a supplementary torque assist in lower-limb powered exoskeletons for persons with paraplegia. In the combined system, also known as a hybrid neuroprosthesis, both FES-assist and the exoskeleton act to generate lower-limb torques to achieve standing and walking functions. Due to this actuator redundancy, we are motivated to optimally allocate FES-assist and exoskeleton torque based on a performance index that penalizes FES overuse to minimize muscle fatigue while also minimizing regulation or tracking errors. Traditional optimal control approaches need a system model to optimize; however, it is often difficult to formulate a musculoskeletal model that accurately predicts muscle responses due to FES. In this paper, we use a novel identification and control structure that contains a recurrent neural network (RNN) and several feedforward neural networks (FNNs). The RNN is trained by supervised learning to identify the system dynamics, while the FNNs are trained by a reinforcement learning method to provide sub-optimal control actions. The output layer of each FNN has its unique activation functions, so that the asymmetric constraint of FES and the symmetric constraint of exoskeleton motor control input can be realized. This new structure is experimentally validated on a seated human participant using a single joint hybrid neuroprosthesis.
Similar content being viewed by others
References
Akpan, V., Hassapis, G.: Nonlinear model identification and adaptive model predictive control using neural networks. ISA Trans. 50(2), 177–194 (2011)
Alibeji, N.A., Molazadeh, V., Dicianno, B.E., Sharma, N.: A control scheme that uses dynamic postural synergies to coordinate a hybrid walking neuroprosthesis: theory and experiments. Front. Neurosci. 12, 159 (2018). https://www.frontiersin.org/article/10.3389/fnins.2018.00159 (Online)
Alibeji, N.A., Molazadeh, V., Moore-Cligenpeel, F., Sharma, N.: A muscle synergy inspired control design to coordinate functional electrical stimulation and a powered exoskeleton. IEEE Control Syst. Mag. 38, 35–60 (2018) (conditionally accepted)
Alibeji, N., Kirsch, N., Sharma, N.: A muscle synergy-inspired adaptive control scheme for a hybrid walking neuroprosthesis. Front. Bioeng. Biotechnol. 3, 203 (2015)
Alibeji, N., Kirsch, N., Sharma, N.: An adaptive low-dimensional control to compensate for actuator redundancy and fes-induced muscle fatigue in a hybrid neuroprosthesis. Control Eng. Pract. 59, 204–219 (2017)
Anaya, F., Thangavel, P., Yu, H.: Hybrid fes-robotic gait rehabilitation technologies: a review on mechanical design, actuation, and control strategies. Int. J. Intell. Robot. Appl. 2(1), 1–28 (2018)
Bao, X., Dicianno, B., Sharma, N.: Model predictive control of a feedback linearized hybrid neuroprosthetic system with a barrier penalty. J. Comput. Nonlinear Dyn. (2019) (in press)
Bao, X., Sun, Z., Sharma, N.: A recurrent neural network based mpc for a hybrid neuroprosthesis system. In: 2017 IEEE 56th Annual Conference on Decision and Control (CDC), IEEE, Melbourne, 12–15 Dec 2017
Beaufays, F., Wan, E.: Relating real-time backpropagation and backpropagation-through-time: an application of flow graph interreciprocity. Neural Comput. 6(2), 296–306 (1994)
Bickel, C., Gregory, C., Dean, J.: Motor unit recruitment during neuromuscular electrical stimulation: a critical appraisal. Eur. J. Appl. Physiol. 111(10), 2399–2407 (2011)
Chen, Y.Q., Yin, T., Babri, H.A.: A stochastic backpropagation algorithm for training neural networks. In: Proceedings of ICICS, 1997 International Conference on Information, Communications and Signal Processing. Theme: Trends in Information Systems Engineering and Wireless Multimedia Communications (Cat No.97TH8237), vol. 2, pp. 703-707. IEEE, Singapore, 12 Sept 1997
del Ama, A., Gil-Agudo, Á., Pons, J., Moreno, J.: Hybrid FES-robot cooperative control of ambulatory gait rehabilitation exoskeleton. J. Neuroeng. Rehabil. 11(1), 27 (2014)
Dodson, A.: A novel user-controlled assisted standing control system for a hybrid neuroprosthesis, Master’s Thesis, University of Pittsburgh (2018)
Durfee, W.K.: Gait restoration by functional electrical stimulation. Climbing and Walking Robots, pp. 19–26. Springer, Berlin, Heidelberg (2006)
Durfee, W.K., Hausdorff, J.M.: Regulating knee joint position by combining electrical stimulation with a controllable friction brake. Ann. Biomed. Eng. 18(6), 575–596 (1990)
Goldfarb, M., Korkowski, K., Harrold, B., Durfee, W.: Preliminary evaluation of a controlled-brake orthosis for FES-aided gait. IEEE Trans. Neural Syst. Rehabil. Eng. 11(3), 241–248 (2003)
Goodfellow, I., Bengio, Y., Courville, A., Bengio, Y.: Deep Learning, vol. 1. MIT press, Cambridge (2016)
Graichen, K., Käpernick, B.: A real-time gradient method for nonlinear model predictive control. INTECH Open Access Publisher, London (2012)
Grondman, I., Busoniu, L., Lopes, G., Babuska, R.: A survey of actor-critic reinforcement learning: standard and natural policy gradients. IEEE Trans. Syst. Man Cybern. C Appl. Rev. 42(6), 1291–1307 (2012)
Ha, K.H., Murray, S.A., Goldfarb, M.: An approach for the cooperative control of FES with a powered exoskeleton during level walking for persons with paraplegia. IEEE Trans. Neural Syst. Rehabil. Eng. 24(4), 455–466 (2015)
Jagodnik, K.M., Thomas, P.S., Van Den Bogert, A.J., Branicky, M.S., Kirsch, R.F.: Human-like rewards to train a reinforcement learning controller for planar arm movement. IEEE Trans. Human Mach. Syst. 46(5), 723–733 (2016)
Jagodnik, K.M., Thomas, P.S., van den Bogert, A.J., Branicky, M.S., Kirsch, R.F.: Training an actor-critic reinforcement learning controller for arm movement using human-generated rewards. IEEE Trans. Neural Syst. Rehabil. Eng. 25(10), 1892–1905 (2017)
Jordan, M., Rumelhart, D.: Forward models: supervised learning with a distal teacher. Cogn. Sci. 16(3), 307–354 (1992)
Kayacan, E., Kayacan, E., Ramon, H., Saeys, W.: Robust tube-based decentralized nonlinear model predictive control of an autonomous tractor-trailer system. IEEE/ASME Trans. Mechatron. 20(1), 447–456 (2015)
Kirsch, N., Alibeji, N., Fisher, L., Gregory, C., Sharma, N.: A semi-active hybrid neuroprosthesis for restoring lower limb function in paraplegics. Conf Proc IEEE Eng Med Biol Soc. 2014, 2557–2560 (2014). https://doi.org/10.1109/EMBC.2014.6944144
Kirsch, N., Alibeji, N., Sharma, N.: Nonlinear model predictive control of functional electrical stimulation. Control Eng. Pract. 58, 319–331 (2017)
Kirsch, N., Bao, X., Alibeji, N., Dicianno, B., Sharma, N.: Model-based dynamic control allocation in a hybrid neuroprosthesis. IEEE Trans. Neural Syst. Rehabil. Eng. 26(1), 224–232 (2018)
Kobetic, R., Marsolais, B.: Synthesis of paraplegic gait with multichannel functional neuromuscular stimulation. IEEE Trans. Rehabil. Eng. 2(2), 66–79 (1994)
Kobetic, R., To, C., Schnellenberger, J., Audu, M., Bulea, T., Gaudio, R., Pinault, G., Tashman, S., Triolo, R.: Development of hybrid orthosis for standing, walking, and stair climbing after spinal cord injury. J. Rehabil. Res. Dev. 46(3), 447–462 (2009)
Lewis, F., Vrabie, D.: Reinforcement learning and adaptive dynamic programming for feedback control. IEEE Circuits Syst. Mag. 9(3), 32–50 (2009)
Lin, L.J., 1993. Reinforcement learning for robots using neural networks. Ph.D. Thesis, Carnegie Mellon University, Pittsburgh PA. Technical Report CMU-CS-93-103 (1993)
Mayne, D., Kerrigan, E., van Wyk, E., Falugi, P.: Tube-based robust nonlinear model predictive control. Int. J. Robust Nonlinear Control 21(11), 1341–1353 (2011)
McCallum, R.A.: Hidden state and reinforcement learning with instance-based state identification. IEEE Trans. Syst. Man Cybern. Part B 26(3), 464–473 (1996)
Munro, P.: A dual back-propagation scheme for scalar reward learning. In: Ninth Annual Conference of the Cognitive Science Society, pp. 165–176 (1987)
Peters, J., Schaal, S.: Policy gradient methods for robotics. In 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2219-2225. IEEE, Beijing, 9–15 Oct 2006
Popović, D., Stein, R., Oğuztöreli, M., Lebiedowska, M., Jonić, S.: Optimal control of walking with functional electrical stimulation: a computer simulation study. IEEE Trans. Rehabil. Eng. 7(1), 69–79 (1999)
Riener, R., Quintern, J., Schmidt, G.: Biomechanical model of the human knee evaluated by neuromuscular stimulation. J. Biomech. 29, 1157–1167 (1996)
Schaefer, A.M., Schneegass, D., Sterzing, V., Udluft, S.: A neural reinforcement learning approach to gas turbine control. In: 2007 International Joint Conference on Neural Networks, pp. 1691–1696. IEEE (2007)
Schäfer, A.M., Udluft, S., et al.: Solving partially observable reinforcement learning problems with recurrent neural networks. In: Workshop Proceedings of the European Conference on Machine Learning, pp. 71–81 (2005)
Schäfer, A.M., Udluft, S., Zimmermann, H.G.: A recurrent control neural network for data efficient reinforcement learning. In: 2007 IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning, pp. 151–157. IEEE (2007)
Sharma, N., Kirsch, N.A., Alibeji, N.A., Dixon, W.E.: A non-linear control method to compensate for muscle fatigue during neuromuscular electrical stimulation. Front. Robot. AI 4, 68 (2017). https://www.frontiersin.org/article/10.3389/frobt.2017.00068 (Online)
Sharma, N., Stegath, K., Gregory, C.M., Dixon, W.E.: Nonlinear neuromuscular electrical stimulation tracking control of a human limb. IEEE Trans. Neural Syst. Rehabil. Eng. 17(6), 576–584 (2012)
Sharma, N., Mushahwar, V., Stein, R.: Dynamic optimization of FES and orthosis-based walking using simple models. IEEE Trans. Neural Syst. Rehabil. Eng. 22, 114–126 (2014)
Silver, D., Huang, A., Maddison, C., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., et al.: Mastering the game of go with deep neural networks and tree search. Nature 529(7587), 484–489 (2016)
Sutton, R., McAllester, D., Singh, S., Mansour, Y., et al.: Policy gradient methods for reinforcement learning with function approximation. NIPS 99, 1057–1063 (1999)
Vallery, H., Stützle, T., Buss, M., Abel, D.: Control of a hybrid motor prosthesis for the knee joint. IFAC Proc. Vol. 38(1), 76–81 (2005)
Vrabie, D., Vamvoudakis, K.G., Lewis, F.L.: Optimal adaptive control and differential games by reinforcement learning principles, vol. 2. IET Press (2013)
Wang, F.-Y., Zhang, H., Liu, D.: Adaptive dynamic programming: an introduction. IEEE Comput. Intell. Mag. 4(2), 39–47 (2009)
Werbos, P.J.: Backpropagation through time: what it does and how to do it. Proc. IEEE 78(10), 1550–1560 (1990)
Williams, R.: Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn. 8(3–4), 229–256 (1992)
Witten, I.H., Frank, E., Hall, M.A., Pal, C.J.: Data Mining: Practical Machine Learning Tools and Techniques. Morgan Kaufmann, Burlington (2016)
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Bao, X., Mao, ZH., Munro, P. et al. Sub-optimally solving actuator redundancy in a hybrid neuroprosthetic system with a multi-layer neural network structure. Int J Intell Robot Appl 3, 298–313 (2019). https://doi.org/10.1007/s41315-019-00100-8
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s41315-019-00100-8