Skip to main content
SpringerLink
Log in

A Bio-Inspired Integration Model of Basal Ganglia and Cerebellum for Motion Learning of a Musculoskeletal Robot

  • Published:
Journal of Systems Science and Complexity Aims and scope Submit manuscript

Abstract

It is a significant research direction for highly complex musculoskeletal robots that how to develop the ability of motion learning and generalization. The cooperations of multiple brain regions are crucial to improving motion performance. Inspired by the neural mechanisms of structures, functions, and interconnections of basal ganglia and cerebellum, a biologically inspired integration model for motor learning of musculoskeletal robots is proposed. Based on the neural characteristics of the basal ganglia, the basal ganglia actor network, which mainly simulates the dorsal striatum, outputs motion commands, and the basal ganglia critic network, which simulates the ventral striatum, estimates action-state values. Their network parameters are updated using the soft actor-critic method. Based on the sensorimotor prediction mechanism of the cerebellum, the cerebellum network evaluates the state feature extraction quality of the basal ganglia actor network and then updates the weights of its feature layer. This learning method is proven to converge to the optimal policy. Furthermore, drawing on the mechanism of dopaminergic dynamic regulation in the basal ganglia, the adaptive adjustment of target entropy and the dopaminergic experience replay are proposed to further improve the integration model, which contributes to the exploration-exploitation trade-off of motor learning. The bio-inspired integration model is validated on a musculoskeletal system. Experimental results indicate that this model can effectively control the musculoskeletal robot to accomplish the motion task from random starting locations to random target positions with high precision and robustness.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Duan G and Liu G P, Attitude and orbit optimal control of combined spacecraft via a fully-actuated system approach, Journal of Systems Science & Complexity, 2022, 35(2): 623–640.

    Article  MathSciNet  Google Scholar 

  2. Hu Y, Guo J, Meng W, et al., Longitudinal control for balloon-borne launched solar powered uavs in near-space, Journal of Systems Science & Complexity, 2022, 35(2): 802–819.

    Article  Google Scholar 

  3. Kong X, Xia Y, Hu R, et al., Trajectory tracking control for under-actuated hovercraft using differential flatness and reinforcement learning-based active disturbance rejection control, Journal of Systems Science & Complexity, 2022, 35(2): 502–521.

    Article  MathSciNet  Google Scholar 

  4. Wang B, Zhang Y, and Zhang W, A composite adaptive fault-tolerant attitude control for a quadrotor UAV with multiple uncertainties, Journal of Systems Science & Complexity, 2022, 35(1): 81–104.

    Article  MathSciNet  Google Scholar 

  5. Qiao H, Chen J, and Huang X, A survey of brain-inspired intelligent robots: Integration of vision, decision, motion control, and musculoskeletal systems, IEEE Transactions on Cybernetics, 2021, 52(10): 11267–11280.

    Article  Google Scholar 

  6. Qiao H, Wu Y, Zhong S, et al., Brain-inspired intelligent robotics: Theoretical analysis and systematic application, Machine Intelligence Research, 2023, 20(1): 1–18.

    Article  Google Scholar 

  7. Qiao H, Zhong S, Chen Z, et al., Improving performance of robots using human-inspired approaches: A survey, Science China Information Sciences, 2022, 65(12): 221201.

    Article  Google Scholar 

  8. Kurumaya S, Suzumori K, Nabae H, et al., Musculoskeletal lower-limb robot driven by multifilament muscles, Robomech Journal, 2016, 3: 1–15.

    Article  Google Scholar 

  9. Wittmeier S, Alessandro C, Bascarevic N, et al., Toward anthropomimetic robotics: Development, simulation, and control of a musculoskeletal torso, Artificial Life, 2013, 19(1): 171–193.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Asano Y, Okada K, and Inaba M, Design principles of a human mimetic humanoid: Humanoid platform to study human intelligence and internal body system, Science Robotics, 2017, 2(13): eaaq0899.

    Article  PubMed  Google Scholar 

  11. Narioka K and Hosoda K, Motor development of an pneumatic musculoskeletal infant robot, 2011 IEEE International Conference on Robotics and Automation, Shanghai, 2011, 963–968.

  12. Boblan I and Schulz A, A humanoid muscle robot torso with biologically inspired construction, ISR 2010 (41st International Symposium on Robotics) and ROBOTIK 2010 (6th German Conference on Robotics), Munich, 2010, 1–6.

  13. Yip M C and Niemeyer G, High-performance robotic muscles from conductive nylon sewing thread, 2015 IEEE International Conference on Robotics and Automation (ICRA), Seattle, 2015, 2313–2318.

  14. Wu Y, Chen J, and Qiao H, Anti-interference analysis of bio-inspired musculoskeletal robotic system, Neurocomputing, 2021, 436: 114–125.

    Article  Google Scholar 

  15. Zhong S, Zhang J, and Nie X, Redundancy reduction of musculoskeletal model for robots with group sparse neural network, 2021 6th International Conference on Control and Robotics Engineering (ICCRE), Beijing, 2021, 39–43.

  16. Zhong S, Chen J, Niu X, et al., Reducing redundancy of musculoskeletal robot with convex hull vertexes selection, IEEE Transactions on Cognitive and Developmental Systems, 2019, 12(3): 601–617.

    Article  Google Scholar 

  17. Thelen D G, Anderson F C, and Delp S L, Generating dynamic simulations of movement using computed muscle control, Journal of Biomechanics, 2003, 36(3): 321–328.

    Article  PubMed  Google Scholar 

  18. Jantsch M, Wittmeier S, Dalamagkidis K, et al., Computed muscle control for an anthro-pomimetic elbow joint, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, Vilamoura-Algarve, 2012, 2192–2197.

  19. Stanev D and Moustakas K, Simulation of constrained musculoskeletal systems in task space, IEEE Transactions on Biomedical Engineering, 2017, 65(2): 307–318.

    PubMed  Google Scholar 

  20. Jantsch M, Wittmeier S, Dalamagkidis K, et al., Adaptive neural network dynamic surface control: An evaluation on the musculoskeletal robot anthrob, 2015 IEEE International Conference on Robotics and Automation (ICRA), Seattle, 2015, 4347–4352.

  21. Li J, Li Z, Li X, et al., Skill learning strategy based on dynamic motion primitives for humanrobot cooperative manipulation, Transactions on Cognitive and Developmental Systems, 2020, 13(1): 105–117.

    Article  Google Scholar 

  22. Lu Z, Wang N, Li M, et al., Incremental motor skill learning and generalization from human dynamic reactions based on dynamic movement primitives and fuzzy logic system, IEEE Transactions on Fuzzy Systems, 2021, 30(6): 1506–1515.

    Article  Google Scholar 

  23. Zeng C, Su H, Li Y, et al., An approach for robotic leaning inspired by biomimetic adaptive control, IEEE Transactions on Industrial Informatics, 2021, 18(3): 1479–1488.

    Article  ADS  Google Scholar 

  24. Li Z, Huang Z, He W, et al., Adaptive impedance control for an upper limb robotic exoskeleton using biological signals, IEEE Transactions on Industrial Electronics, 2016, 64(2): 1664–1674.

    Article  Google Scholar 

  25. Li Z, Li X, Li Q, et al., Human-in-the-loop control of soft exosuits using impedance learning on different terrains, IEEE Transactions on Robotics, 2022, 38(5): 2979–2993.

    Article  Google Scholar 

  26. Lillicrap T P, Hunt J J, Pritzel A, et al., Continuous control with deep reinforcement learning, 2015, arXiv: 1509.02971.

  27. Schulman J, Levine S, Abbeel P, et al., Trust region policy optimization, Proceedings of the International Conference on Machine Learning, PMLR, 2015, 37: 1889–1897.

    Google Scholar 

  28. Schulman J, Wolski F, Dhariwal P, et al., Proximal policy optimization algorithms, 2017, arXiv: 1707.06347.

  29. Kidzmski L, Mohanty S P, Ong C F, et al., Learning to run challenge solutions: Adapting reinforcement learning methods for neuromusculoskeletal environments, The NIPS’17 Competition: Building Intelligent Systems, Springer, Cham, 2018, 121–153.

    Chapter  Google Scholar 

  30. Kidzmski L, Ong C, Mohanty S P, et al., Artificial intelligence for prosthetics: Challenge solutions, The NeurIPS’ 18 Competition: From Machine Learning to Intelligent Conversations, Springer, Cham, 2020, 69–128.

    Chapter  Google Scholar 

  31. Chen J, Chen Z, Yao C, et al., Neural manifold modulated continual reinforcement learning for musculoskeletal robots, IEEE Transactions on Cognitive and Developmental Systems, 2022, DOI: https://doi.org/10.1109/TCDS.2022.3231055.

  32. Chen J and Qiao H, Motor-cortex-like recurrent neural network and multitask learning for the control of musculoskeletal systems, IEEE Transactions on Cognitive and Developmental Systems, 2020, 14(2): 424–436.

    Article  Google Scholar 

  33. Chen J and Qiao H, Muscle-synergies-based neuromuscular control for motion learning and generalization of a musculoskeletal system, IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2020, 51(6): 3993–4006.

    Article  Google Scholar 

  34. Wang X, Chen J, and Wu W, Motion learning for musculoskeletal robots based on cortex-inspired motor primitives and modulation, IEEE Transactions on Cognitive and Developmental Systems, 2023, DOI: https://doi.org/10.1109/TCDS.2023.3293097.

  35. Zhang J, Chen J, Wu W, et al., A cerebellum-inspired prediction and correction model for motion control of a musculoskeletal robot, IEEE Transactions on Cognitive and Developmental Systems, 2023, 15(3): 1209–1223.

    Article  Google Scholar 

  36. Zhong S, Chen Z, and Zhou J, Structure transforming for constructing constraint force field in musculoskeletal robot, Assembly Automation, 2021, 42(2): 169–180.

    Article  Google Scholar 

  37. Zhong S L and Wu W, Motion learning and generalization of musculoskeletal robot using gain primitives, IEEE Transactions on Automation Science and Engineering, 2023, DOI: https://doi.org/10.1109/TASE.2023.3249228.

  38. Zhou J, Zhong S, and Wu W, Hierarchical motion learning for goal-oriented movements with speed-accuracy tradeoff of a musculoskeletal system, IEEE Transactions on Cybernetics, 2021, 52(11): 11453–11466.

    Article  Google Scholar 

  39. Joel D, Niv Y, and Ruppin E, Actor-critic models of the basal ganglia: New anatomical and computational perspectives, Neural Networks, 2002, 15(4–6): 535–547.

    Article  PubMed  Google Scholar 

  40. Kaplan A, Mizrahi-Kliger A D, Israel Z, et al., Dissociable roles of ventral pallidum neurons in the basal ganglia reinforcement learning network, Nature Neuroscience, 2020, 23(4): 556–564.

    Article  PubMed  CAS  Google Scholar 

  41. Takahashi Y, Schoenbaum G, and Niv Y, Silencing the critics: Understanding the effects of cocaine sensitization on dorsolateral and ventral striatum in the context of an actor/critic model, Frontiers in Neuroscience, 2008, 2(1): 86–99.

    Article  PubMed  PubMed Central  Google Scholar 

  42. Caligiore D, Arbib M A, Miall R C, et al., The super-learning hypothesis: Integrating learning processes across cortex, cerebellum and basal ganglia, Neuroscience & Biobehavioral Reviews, 2019, 100: 19–34.

    Article  Google Scholar 

  43. Dasgupta S, Wörgötter F, and Manoonpong P, Neuromodulatory adaptive combination of correlation-based learning in cerebellum and reward-based learning in basal ganglia for goal-directed behavior control, Frontiers in Neural Circuits, 2014, 8: 126.

    Article  PubMed  PubMed Central  Google Scholar 

  44. Wang D, Chen S, Hu Y, et al., Behavior decision of mobile robot with a neurophysiologically motivated reinforcement learning model, IEEE Transactions on Cognitive and Developmental Systems, 2020, 14(1): 219–233.

    Article  Google Scholar 

  45. Wang D, Hu Y, and Ma T, Mobile robot navigation with the combination of supervised learning in cerebellum and reward-based learning in basal ganglia, Cognitive Systems Research, 2020, 59: 1–14.

    Article  CAS  Google Scholar 

  46. Zhu J, Wang D, and Si J, Flexible behavioral decision making of mobile robot in dynamic environment, IEEE Transactions on Cognitive and Developmental Systems, 2022, 15(1): 134–149.

    Article  Google Scholar 

  47. Ruan X, Chen J, and Yu N, Thalamic cooperation between the cerebellum and basal ganglia with a new tropism-based action-dependent heuristic dynamic programming method, Neurocomputing, 2012, 93: 27–40.

    Article  Google Scholar 

  48. Calabresi P, Picconi B, Tozzi A, et al., Direct and indirect pathways of basal ganglia: A critical reappraisal, Nature Neuroscience, 2014, 17(8): 1022–1030.

    Article  PubMed  CAS  Google Scholar 

  49. Jin X, Tecuapetla F, and Costa R M, Basal ganglia subcircuits distinctively encode the parsing and concatenation of action sequences, Nature Neuroscience, 2014, 17(3): 423–430.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  50. Gazzaniga M, Ivry R, and Mangun G, Cognitive Neuroscience: The Biology of the Mind, W.W. Norton & Company, New York, 2019.

    Google Scholar 

  51. Stephenson-Jones M, Yu K, Ahrens S, et al., A basal ganglia circuit for evaluating action outcomes, Nature, 2016, 539(7628): 289–293.

    Article  PubMed  PubMed Central  ADS  Google Scholar 

  52. Abadia I, Naveros F, Ros E, et al., A cerebellar-based solution to the nondeterministic time delay problem in robotic control, Science Robotics, 2021, 6(58): eabf2756.

    Article  PubMed  Google Scholar 

  53. Bouvier G, Aljadeff J, Clopath C, et al., Cerebellar learning using perturbations, Elife, 2018, 7: e31599.

    Article  PubMed  PubMed Central  Google Scholar 

  54. Manto M, Bower J M, Conforto A B, et al., Consensus paper: Roles of the cerebellum in motor control — The diversity of ideas on cerebellar involvement in movement, The Cerebellum, 2012, 11: 457–487.

    Article  PubMed  Google Scholar 

  55. Bostan A C, Dum R P, and Strick P L, The basal ganglia communicate with the cerebellum, Proceedings of the National Academy of Sciences, 2010, 107(18): 8452–8456.

    Article  ADS  CAS  Google Scholar 

  56. Bostan A C and Strick P L, The basal ganglia and the cerebellum: Nodes in an integrated network, Nature Reviews Neuroscience, 2018, 19(6): 338–350.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  57. Hoshi E, Tremblay L, Féger J, et al., The cerebellum communicates with the basal ganglia, Nature Neuroscience, 2005, 8(11): 1491–1493.

    Article  PubMed  CAS  Google Scholar 

  58. Wagner M J, Kim T H, Savall J, et al., Cerebellar granule cells encode the expectation of reward, Nature, 2017, 544(7648): 96–100.

    Article  PubMed  PubMed Central  ADS  CAS  Google Scholar 

  59. Chen C H, Fremont R, Arteaga-Bracho E E, et al., Short latency cerebellar modulation of the basal ganglia, Nature Neuroscience, 17(12): 1767–1775.

  60. Yoshida J, Onate M, Khatami L, et al., Cerebellar contributions to the basal ganglia influence motor coordination, reward processing, and movement vigor, Journal of Neuroscience, 2022, 42(45): 8406–8415.

    Article  PubMed  CAS  Google Scholar 

  61. Ohmae S and Medina J F, Climbing fibers encode a temporal-difference prediction error during cerebellar learning in mice, Nature Neuroscience, 2015, 18(12): 1798–1803.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  62. Haarnoja T, Zhou A, Abbeel P, Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, International Conference on Machine Learning, PMLR, Stockholm, 2018, 1861–1870.

  63. Haarnoja T, Zhou A, Hartikainen K, et al., Soft actor-critic algorithms and applications, 2018, arXiv: 1812.05905.

  64. Humphries M D, Khamassi M, and Gurney K, Dopaminergic control of the exploration-exploitation trade-off via the basal ganglia, Frontiers in Neuroscience, 2012, 6: 16922.

    Article  Google Scholar 

  65. Budzillo A, Duffy A, Miller K E, et al., Dopaminergic modulation of basal ganglia output through coupled excitation-inhibition, Proceedings of the National Academy of Sciences, 2017, 114(22): 5713–5718.

    Article  ADS  CAS  Google Scholar 

  66. Zhao F, Zeng Y, Wang G, et al., A brain-inspired decision making model based on top-down biasing of prefrontal cortex to basal ganglia and its application in autonomous UAV explorations, Cognitive Computation, 2018, 10: 296–306.

    Article  Google Scholar 

  67. Fujimoto S, Hoof H, and Meger D, Addressing function approximation error in actor-critic methods, International Conference on Machine Learning, PMLR, Stockholm, 2018, 1587–1596.

  68. Haarnoja T, Ha S, Zhou A, et al., Learning to walk via deep reinforcement learning, 2018, arXiv: 1812.11103.

  69. Mnih V, Kavukcuoglu K, Silver D, et al., Human-level control through deep reinforcement learning, Nature, 2015, 518(7540): 529–533.

    Article  PubMed  ADS  CAS  Google Scholar 

  70. Katharopoulos A and Fleuret F, Not all samples are created equal: Deep learning with importance sampling, International Conference on Machine Learning, PMLR, Stockholm, 2018, 2525–2534.

  71. Wang C, Wu Y, Vuong Q, et al., Striving for simplicity and performance in off-policy drl: Output normalization and non-uniform sampling, International Conference on Machine Learning, PMLR, 2020, 10070–10080.

  72. Constantinidis C and Klingberg T, The neuroscience of working memory capacity and training, Nature Reviews Neuroscience, 2016, 17(7): 438–449.

    Article  PubMed  CAS  Google Scholar 

  73. Cools R and D’Esposito M, Inverted-u-shaped dopamine actions on human working memory and cognitive control, Biological Psychiatry, 2011, 69(12): e113–e125.

    Article  PubMed  PubMed Central  CAS  Google Scholar 

  74. Todorov E, Erez T, and Tassa Y, Mujoco: A physics engine for model-based control, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, Vilamoura-Algarve, 2012, 5026–5033.

  75. Elliott D, Hansen S, Grierson L E, et al., Goal-directed aiming: Two components but multiple processes, Psychological Bulletin, 2010, 136(6): 1023–1044.

    Article  PubMed  Google Scholar 

  76. Elliott D, Lyons J, Hayes S J, et al., The multiple process model ofgoal-directed reaching revisited, Neuroscience & Biobehavioral Reviews, 2017, 72: 95–110.

    Article  Google Scholar 

  77. Huang X, Wu W, and Qiao H, Connecting model-based and model-free control with emotion modulation in learning systems, IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2019, 51(8): 4624–4638.

    Article  Google Scholar 

  78. Liu Z, Lu Z, Zhao Z, et al., Single parameter adaptive neural network control for multi-agent deployment with prescribed tracking performance, Automatica, 2023, 156: 111207.

    Article  MathSciNet  Google Scholar 

  79. Li Z, Li G, Wu X, et al., Asymmetric cooperation control of dual-arm exoskeletons using human collaborative manipulation models, IEEE Transactions on Cybernetics, 2021, 52(11): 12126–12139.

    Article  Google Scholar 

  80. Wang Y, Li H, Zhao Y, et al., A fast coordinated motion planning method for dual-arm robot based on parallel constrained DDP, IEEE/ASME Transactions on Mechatronics, 2023, DOI: https://doi.org/10.1109/TMECH.2023.3323798.

Download references

Author information

Authors and Affiliations

  1. State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China

    Jinhan Zhang, Jiahao Chen, Shanlin Zhong & Hong Qiao

  2. School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, 100049, China

    Jinhan Zhang, Jiahao Chen, Shanlin Zhong & Hong Qiao

Authors
  1. Jinhan Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jiahao Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shanlin Zhong
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hong Qiao
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hong Qiao.

Ethics declarations

The authors declare no conflict of interest.

Additional information

This research was supported by Major Project of Science and Technology Innovation 2030 Brain Science and Brain-Inspired Intelligence under Grant No. 2021ZD0200408, the National Natural Science Foundation of China under Grant Nos. 62203439 and 62203443, and Major program of the National Natural Science Foundation of China under Grant Nos. T2293720, T2293723, and T2293724.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Zhang, J., Chen, J., Zhong, S. et al. A Bio-Inspired Integration Model of Basal Ganglia and Cerebellum for Motion Learning of a Musculoskeletal Robot. J Syst Sci Complex 37, 82–113 (2024). https://doi.org/10.1007/s11424-024-3414-7

Download citation

  • Received:

  • Revised:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11424-024-3414-7

Keywords

Search

Navigation