Abstract
Behaviour selection has been an active research topic for robotics, in particular in the field of human–robot interaction. For a robot to interact autonomously and effectively with humans, the coupling between techniques for human activity recognition and robot behaviour selection is of paramount importance. However, most approaches to date consist of deterministic associations between the recognised activities and the robot behaviours, neglecting the uncertainty inherent to sequential predictions in real-time applications. In this paper, we address this gap by presenting an initial neurorobotics model that embeds, in a simulated robot, computational models of parts of the mammalian brain that resembles neurophysiological aspects of the basal ganglia–thalamus–cortex (BG–T–C) circuit, coupled with human activity recognition techniques. A robotics simulation environment was developed for assessing the model, where a mobile robot accomplished tasks by using behaviour selection in accordance with the activity being performed by the inhabitant of an intelligent home. Initial results revealed that the initial neurorobotics model is advantageous, especially considering the coupling between the most accurate activity recognition approaches and the computational models of more complex animals.
Similar content being viewed by others
Data availability
The HWU-USP activities dataset is available at Data Dryad (Ranieri et al. 2021). All code employed in this paper is available at Github, under the following repositories: Activity recognition framework: https://github.com/cmranieri/Deep-Activity-Recognition, Bioinspired computational model and decoder: https://github.com/cmranieri/Bioinspired-behaviour, Robot simulation: https://github.com/cmranieri/robot-simulation
References
Abiri R, Borhani S, Sellers EW, Jiang Y, Zhao X (2019) A comprehensive review of EEG-based brain-computer interface paradigms. J Neural Eng 16(1):011001
Amato G, Bacciu D, Broxvall M, Chessa S, Coleman S, Di Rocco M, Dragone M, Gallicchio C, Gennaro C, Lozano H, McGinnity TM, Micheli A, Ray AK, Renteria A, Saffiotti A, Swords D, Vairo C, Vance P (2015) Robotic ubiquitous cognitive ecology for smart homes. J Intell Robot Syst Theory Appl. https://doi.org/10.1007/s10846-015-0178-2
Ashry S, Ogawa T, Gomaa W (2020) CHARM-deep: continuous human activity recognition model based on deep neural network using IMU sensors of smartwatch. IEEE Sens J 20(15):8757–8770. https://doi.org/10.1109/JSEN.2020.2985374
Assembly UG (1948) Universal declaration of human rights. UN General Assembly
Bacciu D, Di Rocco M, Dragone M, Gallicchio C, Micheli A, Saffiotti A (2019) An ambient intelligence approach for learning in smart robotic environments. Comput Intell. https://doi.org/10.1111/coin.12233
Bahuguna J, Weidel P, Morrison A (2018) Exploring the role of striatal D1 and D2 medium spiny neurons in action selection using a virtual robotic framework. Eur J Neurosci 49(6):737–753. https://doi.org/10.1111/ejn.14021
Bariselli S, Fobbs WC, Creed MC, Kravitz AV (2019) A competitive model for striatal action selection. Brain Res. https://doi.org/10.1016/j.brainres.2018.10.009
Caine KE, Rogers WA, Fisk AD (2005) Privacy perceptions of an aware home with visual sensing devices. Proc Hum Factors Ergon Soc Annu Meet 49(21):1856–1858. https://doi.org/10.1177/154193120504902108
Calvaresi D, Cesarini D, Sernani P, Marinoni M, Dragoni AF, Sturm A (2017) Exploring the ambient assisted living domain: a systematic review. J Ambient Intell Humaniz Comput 8(2):239–257
Chen K, Zhang D, Yao L, Guo B, Yu Z, Liu Y (2021) Deep learning for sensor-based human activity recognition: overview, challenges, and opportunities. ACM Comput Surv (CSUR) 54(4):1–40
Dawson TM, Golde TE, Lagier-Tourenne C (2018) Animal models of neurodegenerative diseases. Nat Neurosci 21(10):1370–1379. https://doi.org/10.1038/s41593-018-0236-8
Drakopoulos F, Baby D, Verhulst S (2021) A convolutional neural-network framework for modelling auditory sensory cells and synapses. Commun Biol 4(1):1–17
Dura-Bernal S, Suter BA, Gleeson P, Cantarelli M, Quintana A, Rodriguez F, Kedziora DJ, Chadderdon GL, Kerr CC, Neymotin SA, McDougal RA, Hines M, Shepherd GM, Lytton WW (2019) NetPyNE, a tool for data-driven multiscale modeling of brain circuits. eLife 8:e44494. https://doi.org/10.7554/eLife.44494
Fernandes Junior FE, Yang G, Do HM, Sheng W (2016) Detection of privacy-sensitive situations for social robots in smart homes. In: Automation science and engineering (CASE), 2016 IEEE international conference on, pp 727–732. IEEE. https://doi.org/10.1109/COASE.2016.7743474
Garcia FA, Ranieri CM, Romero RAF (2019) Temporal approaches for human activity recognition using inertial sensors. In: Proceedings—2019 Latin American robotics symposium, 2019 Brazilian symposium on robotics and 2019 workshop on robotics in education, LARS/SBR/WRE 2019, pp 121–125. Institute of electrical and electronics engineers Inc. https://doi.org/10.1109/LARS-SBR-WRE48964.2019.00029
Georgievski I, Nguyen TA, Nizamic F, Setz B, Lazovik A, Aiello M (2017) Planning meets activity recognition: service coordination for intelligent buildings. Pervasive Mob Comput 38:110–139. https://doi.org/10.1016/j.pmcj.2017.02.008
Girard B, Tabareau N, Pham QC, Berthoz A, Slotine JJ (2008) Where neuroscience and dynamic system theory meet autonomous robotics: a contracting basal ganglia model for action selection. Neural Netw 21(4):628–641. https://doi.org/10.1016/j.neunet.2008.03.009
Grisetti G, Stachniss C, Burgard W (2007) Improved techniques for grid mapping with Rao-Blackwellized particle filters. IEEE Trans Robot 23(1):34–46
Halje P, Brys I, Mariman JJ, Da Cunha C, Fuentes R, Petersson P (2019) Oscillations in cortico-basal ganglia circuits: implications for Parkinson’s disease and other neurologic and psychiatric conditions. J Neurophysiol 122(1):203–231. https://doi.org/10.1152/jn.00590.2018
Haykin SS (2008) Neural networks: a comprehensive foundation, 3rd edn. Pearson, London
Herath S, Harandi M, Porikli F (2017) Going deeper into action recognition: a survey. Image Vis Comput 60:4–21. https://doi.org/10.1016/j.imavis.2017.01.010
Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735
Hwu TJ, Krichmar JL (2022) Neurorobotics: neuroscience and robots. In: Cangelosi A, Asada M (eds) Cognitive robotics, Chap 2. MIT Press, Cambridge, pp 19–40
Imran J, Raman B (2020) Evaluating fusion of RGB-D and inertial sensors for multimodal human action recognition. J Ambient Intell Humaniz Comput 11(1):189–208. https://doi.org/10.1007/s12652-019-01239-9
Kita H, Kita T (2011) Cortical stimulation evokes abnormal responses in the dopamine-depleted rat basal ganglia. J Neurosci 31(28):10311–10322. https://doi.org/10.1523/JNEUROSCI.0915-11.2011
Koenig N, Howard A (2004) Design and use paradigms for gazebo, an open-source multi-robot simulator. In: 2004 IEEE/RSJ international conference on intelligent robots and systems (IROS)(IEEE Cat. No. 04CH37566), vol 3, pp 2149–2154. IEEE
Könings B, Schaub F, Weber M (2016) Privacy and trust in ambient intelligent environments. In: Next generation intelligent environments. Springer, pp 133–164
Koprich JB, Kalia LV, Brotchie JM (2017) Animal models of alpha-synucleinopathy for Parkinson disease drug development. Nat Rev Neurosci 18(9):515–529. https://doi.org/10.1038/nrn.2017.75
Krichmar JL (2018) Neurorobotics-a thriving community and a promising pathway toward intelligent cognitive robots. Front Neurorobotics 12:42
Kumaravelu K, Brocker DT, Grill WM (2016) A biophysical model of the cortex-basal ganglia-thalamus network in the 6-OHDA lesioned rat model of Parkinson’s disease. J Comput Neurosci 40(2):207–229. https://doi.org/10.1007/s10827-016-0593-9
Lakshmanan V, Robinson S, Munn M (2020) Machine learning design patterns. O’Reilly Media, Sebastopol
Lánský P, Rodriguez R, Sacerdote L (2004) Mean instantaneous firing frequency is always higher than the firing rate. Neural Comput 16(3):477–489. https://doi.org/10.1162/089976604772744875
Li J, Li Z, Chen F, Bicchi A, Sun Y, Fukuda T (2019) Combined sensing, cognition, learning, and control for developing future neuro-robotics systems: a survey. IEEE Trans Cognit Dev Syst 11(2):148–161. https://doi.org/10.1109/TCDS.2019.2897618
Li Z, Liu F, Yang W, Peng S, Zhou J (2021) A survey of convolutional neural networks: analysis, applications, and prospects. IEEE Trans Neural Netw Learn Syst. https://doi.org/10.1109/TNNLS.2021.3084827
Liang Y, Yan Z, Zhang Q, Liang H, Ji X, Liu Y, Liu R (2019) A decision-making model based on basal ganglia account of action prediction. In: IEEE international conference on robotics and biomimetics, ROBIO 2019. Institute of electrical and electronics engineers Inc., pp 1705–1710 https://doi.org/10.1109/ROBIO49542.2019.8961538
Liénard J, Girard B (2014) A biologically constrained model of the whole basal ganglia addressing the paradoxes of connections and selection. J Comput Neurosci 36(3):445–468. https://doi.org/10.1007/s10827-013-0476-2
Li K, Wu J, Zhao X, Tan M (2019) Real-time human-robot interaction for a service robot based on 3D human activity recognition and human-mimicking decision mechanism. In: 8th annual IEEE international conference on cyber technology in automation, control and intelligent systems, CYBER 2018, pp 498–503. Institute of electrical and electronics engineers Inc. https://doi.org/10.1109/CYBER.2018.8688272
Lu Y, Velipasalar S (2019) Autonomous human activity classification from wearable multi-modal sensors. IEEE Sens J 19(23):11403–11412. https://doi.org/10.1109/JSEN.2019.2934678
Luu DK, Nguyen AT, Jiang M, Xu J, Drealan MW, Cheng J, Keefer EW, Zhao Q, Yang Z (2021) Deep learning-based approaches for decoding motor intent from peripheral nerve signals. Front Neurosci 15:667907
Ma CY, Chen MH, Kira Z, AlRegib G (2019) TS-LSTM and temporal-inception: Exploiting spatiotemporal dynamics for activity recognition. Signal Processing: Image Communication 71:76–87. https://doi.org/10.1016/j.image.2018.09.003
Markowitz JE, Gillis WF, Beron CC, Neufeld SQ, Robertson K, Bhagat ND, Peterson RE, Peterson E, Hyun M, Linderman SW, Sabatini BL, Datta SR (2018) The striatum organizes 3D behavior via moment-to-moment action selection. Cell 174(1):44–58. https://doi.org/10.1016/j.cell.2018.04.019
McGregor MM, Nelson AB (2019) Circuit mechanisms of Parkinson’s disease. Neuron 101(6):1042–1056. https://doi.org/10.1016/j.neuron.2019.03.004
Merk T, Peterson V, Köhler R, Haufe S, Richardson RM, Neumann WJ (2022) Machine learning based brain signal decoding for intelligent adaptive deep brain stimulation. Exp Neurol. https://doi.org/10.1016/j.expneurol.2022.113993
Mojarad R, Attal F, Chibani A, Fiorini SR, Amirat Y (2018) Hybrid approach for human activity recognition by ubiquitous robots. In: IEEE international conference on intelligent robots and systems. Institute of electrical and electronics engineers Inc., pp 5660–5665 https://doi.org/10.1109/IROS.2018.8594173
Mora-Sánchez A, Dreyfus G, Vialatte FB (2019) Scale-free behaviour and metastable brain-state switching driven by human cognition, an empirical approach. Cogn Neurodyn 13(5):437–452
Mulcahy G, Atwood B, Kuznetsov A (2020) Basal ganglia role in learning rewarded actions and executing previously learned choices: healthy and diseased states. PLoS ONE 15(2):1–26
Oh SL, Hagiwara Y, Raghavendra U, Yuvaraj R, Arunkumar N, Murugappan M, Acharya UR (2018) A deep learning approach for Parkinson’s disease diagnosis from EEG signals. Neural Comput Appl. https://doi.org/10.1007/s00521-018-3689-5
Ordóñez F, Roggen D (2016) Deep convolutional and LSTM recurrent neural networks for multimodal wearable activity recognition. Sensors 16(1):115. https://doi.org/10.3390/s16010115
Pimentel JM, Moioli RC, De Araujo MF, Ranieri CM, Romero RA, Broz F, Vargas PA (2021) Neuro4pd: an initial neurorobotics model of Parkinson’s disease. Front Neurorobotics 88
Prescott TJ, Montes González FM, Gurney K, Humphries MD, Redgrave P (2006) A robot model of the basal ganglia: behavior and intrinsic processing. Neural Netw 19(1):31–61. https://doi.org/10.1016/j.neunet.2005.06.049
Pronin S, Wellacott L, Pimentel J, Moioli RC, Vargas PA (2021) Neurorobotic models of neurological disorders: a mini review. Front Neurorobotics 15:26
Quigley M, Conley K, Gerkey B, Faust J, Foote T, Leibs J, Wheeler R, Ng AY (2009) ROS: an open-source robot operating system. In: ICRA workshop on open source software, vol 3, p 5. Kobe, Japan
Ranieri CM, MacLeod S, Dragone M, Vargas PA, Romero RAF (2021) Human activities with videos, inertial units and ambient sensors. Dryad Digital Repository
Ranieri CM, Nardari G, Pinto AH, Tozadore DC, Romero RA (2018) LARa: a robotic framework for human-robot interaction on indoor environments. In: Proceedings—15th Latin American robotics symposium, 6th Brazilian robotics symposium and 9th workshop on robotics in education, LARS/SBR/WRE 2018. Institute of electrical and electronics engineers Inc., pp 383–389 https://doi.org/10.1109/LARS/SBR/WRE.2018.00074
Ranieri CM, MacLeod S, Dragone M, Vargas PA, Romero RF (2021) Activity recognition for ambient assisted living with videos, inertial units and ambient sensors. Sensors 21(3):768. https://doi.org/10.3390/S21030768
Ranieri CM, Pimentel JM, Romano MR, Elias LA, Romero RA, Lones MA, Araujo MF, Vargas PA, Moioli RC (2021) A data-driven biophysical computational model of Parkinson’s disease based on marmoset monkeys. IEEE Access 9:122548–122567
Ranieri C, Moioli R, Romero R, De Araujo M, De Santana M, Pimentel J, Vargas P (2020) Unveiling Parkinson’s disease features from a primate model with deep neural networks. In: Proceedings of the international joint conference on neural networks. https://doi.org/10.1109/IJCNN48605.2020.9207180
Ranieri C, Vargas P, Romero R (2020) Uncovering human multimodal activity recognition with a deep learning approach. In: Proceedings of the international joint conference on neural networks. https://doi.org/10.1109/IJCNN48605.2020.9207255
Rodriguez Lera FJ, Martín Rico F, Guerrero Higueras AM, Olivera VM (2020) A context-awareness model for activity recognition in robot-assisted scenarios. Expert Syst 37(2):e12481. https://doi.org/10.1111/exsy.12481
Rodríguez-Moreno I, Martínez-Otzeta JM, Sierra B, Rodriguez I, Jauregi E (2019) Video activity recognition: state-of-the-art. Sensors 19(14):3160. https://doi.org/10.3390/s19143160
Rucci M, Bullock D, Santini F (2007) Integrating robotics and neuroscience: brains for robots, bodies for brains. Adv Robot 21(10):1115–1129
Saeidi M, Karwowski W, Farahani FV, Fiok K, Taiar R, Hancock P, Al-Juaid A (2021) Neural decoding of EEG signals with machine learning: a systematic review. Brain Sci 11(11):1525
Selvaggio M, Cognetti M, Nikolaidis S, Ivaldi S, Siciliano B (2021) Autonomy in physical human-robot interaction: a brief survey. IEEE Robot Autom Lett. https://doi.org/10.1109/LRA.2021.3100603
Sen S, Daimi SN, Watanabe K, Takahashi K, Bhattacharya J, Saha G (2020) Switch or stay? automatic classification of internal mental states in bistable perception. Cogn Neurodyn 14(1):95–113
Sharkey A, Sharkey N (2012) Granny and the robots: ethical issues in robot care for the elderly. Ethics Inf Technol 14(1):27–40. https://doi.org/10.1007/s10676-010-9234-6
Steven Eyobu O, Han D (2018) Feature representation and data augmentation for human activity classification based on wearable IMU sensor data using a deep LSTM neural network. Sensors 18(9):2892. https://doi.org/10.3390/s18092892
Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: IEEE conference on computer vision and pattern recognition (CVPR). IEEE, Las Vegas, NV, USA, pp 2818–2826
Tinkhauser G, Pogosyan A, Tan H, Herz DM, Kühn AA, Brown P (2017) Beta burst dynamics in Parkinson’s disease off and on dopaminergic medication. Brain 140(11):2968–2981. https://doi.org/10.1093/brain/awx252
van Albada SJ, Robinson PA (2009) Mean-field modeling of the basal ganglia-thalamocortical system. I. Firing rates in healthy and parkinsonian states. J Theor Biol 257(4):642–663. https://doi.org/10.1016/j.jtbi.2008.12.018
van der Heijden K, Mehrkanoon S (2022) Goal-driven, neurobiological-inspired convolutional neural network models of human spatial hearing. Neurocomputing 470:432–442
Van Der Smagt P, Arbib MA, Metta G (2016) Neurorobotics: from vision to action. In: Springer handbook of robotics. Springer International Publishing, pp 2069–2094
Wojtowytsch S, Weinan E (2020) Can shallow neural networks beat the curse of dimensionality? A mean field training perspective. IEEE Trans Artif Intell 1(2):121–129
Zach C, Pock T, Bischof H (2007) A duality based approach for real-time TV-L 1 optical flow. In: Joint pattern recognition symposium. Springer, pp 214–223
Zahra O, Navarro-Alarcon D, Tolu S (2021) A neurorobotic embodiment for exploring the dynamical interactions of a spiking cerebellar model and a robot arm during vision-based manipulation tasks. Int J Neural Syst 32:2150028
Zeiler MD, Fergus R (2014) Visualizing and understanding convolutional networks. In: European conference on computer vision. Springer, pp 818–833
Zheng P, Kozloski J (2017) Striatal network models of Huntington’s disease dysfunction phenotypes. Front Comput Neurosci 11:70
Funding
This work was funded by the Sao Paulo Research Foundation (FAPESP), grants 2017/02377-5, 2017/01687-0, 2018/25902-0 and 2021/10921-2, and the Neuro4PD project - Royal Society and Newton Fund (NAF/ R2/180773). Moioli acknowledge the support from the Brazilian institutions: INCT INCEMAQ of the CNPq/MCTI, FAPERN, CAPES, FINEP, and MEC. This research was carried out using the computational resources from the CeMEAI funded by FAPESP, grant 2013/07375-0. Additional resources were provided by the Robotics Lab within the ECR, and by the Nvidia Grants program.
Author information
Authors and Affiliations
Contributions
All authors contributed to the design of the experiments. Ranieri provided the specific methods and implementations, performed the experiments and analysed the results. Moioli, Vargas and Romero revised the methods and results presented, contributing to the discussion. Ranieri written the draft of the paper, revised by the other authors.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Consent for publication
All authors agreed to publish this paper in its present form.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Ranieri, C.M., Moioli, R.C., Vargas, P.A. et al. A neurorobotics approach to behaviour selection based on human activity recognition. Cogn Neurodyn 17, 1009–1028 (2023). https://doi.org/10.1007/s11571-022-09886-z
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11571-022-09886-z