Abstract
We present Deep Control – new ML architecture of cortico-striatal brain circuits, which use whole cortical column as a structural element, instead of a singe neuron. Based on this architecture, we present MARTI - new model of human brain, considering neocortex and basal ganglia. This model is designed to implement expedient behavior and is capable to learn and achieve goals in unknown environments. We introduce a novel surprise feeling mechanism, that significantly improves reinforcement learning process through inner rewards. We use OpenAI Gym environment to demonstrate MARTI learning on a single CPU just in several hours.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Shumsky, S.A.: Machine intelligence. Essays on the theory of machine learning and artificial intelligence. RIOR Publishing, Moscow (2019). ISBN 978-5-369-02011-1.
Friston, K.: A theory of cortical responses. Philos. Trans. R. Soc. B: Biol. Sci. 360(1456), 815–836 (2005)
Bastos, A.M., et al.: Canonical microcircuits for predictive coding. Neuron 76(4), 695–711 (2012)
Clark, A.: Surfing Uncertainty: Prediction, Action, and The Embodied Mind. Oxford University Press, Oxford (2015)
Spratling, M.W.: A review of predictive coding algorithms. Brain Cogn. 112, 92–97 (2017)
Hawkins, J., Ahmad, S.: Why neurons have thousands of synapses, a theory of sequence memory in neocortex. Front. Neural Circuits 10, 23 (2016)
Hawkins, J., Ahmad, S., Cui, Y.: A theory of how columns in the neocortex enable learning the structure of the world. Front. Neural Circuits 11, 81 (2017)
Laukien, E., Richard C., Fergal B.: Feynman machine: the universal dynamical systems computer. arXiv preprint arXiv:1609.03971 (2016)
Caligiore, D., et al.: The super-learning hypothesis: integrating learning processes across cortex, cerebellum and basal ganglia. Neurosci. Biobehav. Rev. 100, 19–34 (2019)
Botvinick, M.M.: Hierarchical reinforcement learning and decision making. Curr. Opin. Neurobiol. 22(6), 956–962 (2012)
Pateria, S., et al.: Hierarchical reinforcement learning: a comprehensive survey. ACM Comput. Surv. (CSUR) 54(5), 1–35 (2021)
Sutton, R.S., Precup, D., Singh, S.: Between MDPs and semi-MDPs: a framework for temporal abstraction in reinforcement learning. Artif. Intell. 112(1–2), 181–211 (1999)
Bacon, P.L., H, J., Precup, D.: The option-critic architecture. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 31, no. 1 (2017)
Vezhnevets, A.S, et al.: Feudal networks for hierarchical reinforcement learning. In: International Conference on Machine Learning. PMLR (2017)
Nachum, O., et al.: Data-efficient hierarchical reinforcement learning. arXiv preprint arXiv:1805.08296 (2018)
Brockman, G., et al.: Open AI Gym. arXiv 2016
Marc, B.G., et al.: The arcade learning environment: an evaluation platform for general agents. J. Artif. Intell. Res. 47, 253–279 (2013)
Shumsky, S.A.: Deep structural learning: a new look at reinforcement learning. In: XX Russian Scientific Conference NEUROINFORMATICS 2018. Lectures on Neuroinformatics, pp. 11–43 (2018)
Mnih, V., et al.: Playing Atari with Deep Reinforcement Learning. arXiv 2013
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Ethics declarations
Igor Pivovarov works part time in Moscow Institute of Physics and Technologies, Huawei, Skoltech, Bauman University and IP Laboratories. Sergey Shumsky works part time in Moscow Institute of Physics and Technologies and Bauman University. However, the whole scope of current work was made by authors solely in free time without any support or participation of any entities.
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Pivovarov, I., Shumsky, S. (2023). MARTI-4: New Model of Human Brain, Considering Neocortex and Basal Ganglia – Learns to Play Atari Game by Reinforcement Learning on a Single CPU. In: Goertzel, B., Iklé, M., Potapov, A., Ponomaryov, D. (eds) Artificial General Intelligence. AGI 2022. Lecture Notes in Computer Science(), vol 13539. Springer, Cham. https://doi.org/10.1007/978-3-031-19907-3_7
Download citation
DOI: https://doi.org/10.1007/978-3-031-19907-3_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-19906-6
Online ISBN: 978-3-031-19907-3
eBook Packages: Computer ScienceComputer Science (R0)