Skip to main content
Log in

A neurorobotics approach to behaviour selection based on human activity recognition

  • Research Article
  • Published:
Cognitive Neurodynamics Aims and scope Submit manuscript

Abstract

Behaviour selection has been an active research topic for robotics, in particular in the field of human–robot interaction. For a robot to interact autonomously and effectively with humans, the coupling between techniques for human activity recognition and robot behaviour selection is of paramount importance. However, most approaches to date consist of deterministic associations between the recognised activities and the robot behaviours, neglecting the uncertainty inherent to sequential predictions in real-time applications. In this paper, we address this gap by presenting an initial neurorobotics model that embeds, in a simulated robot, computational models of parts of the mammalian brain that resembles neurophysiological aspects of the basal ganglia–thalamus–cortex (BG–T–C) circuit, coupled with human activity recognition techniques. A robotics simulation environment was developed for assessing the model, where a mobile robot accomplished tasks by using behaviour selection in accordance with the activity being performed by the inhabitant of an intelligent home. Initial results revealed that the initial neurorobotics model is advantageous, especially considering the coupling between the most accurate activity recognition approaches and the computational models of more complex animals.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data availability

The HWU-USP activities dataset is available at Data Dryad (Ranieri et al. 2021). All code employed in this paper is available at Github, under the following repositories: Activity recognition framework: https://github.com/cmranieri/Deep-Activity-Recognition, Bioinspired computational model and decoder: https://github.com/cmranieri/Bioinspired-behaviour, Robot simulation: https://github.com/cmranieri/robot-simulation

References

  • Abiri R, Borhani S, Sellers EW, Jiang Y, Zhao X (2019) A comprehensive review of EEG-based brain-computer interface paradigms. J Neural Eng 16(1):011001

    Article  PubMed  Google Scholar 

  • Amato G, Bacciu D, Broxvall M, Chessa S, Coleman S, Di Rocco M, Dragone M, Gallicchio C, Gennaro C, Lozano H, McGinnity TM, Micheli A, Ray AK, Renteria A, Saffiotti A, Swords D, Vairo C, Vance P (2015) Robotic ubiquitous cognitive ecology for smart homes. J Intell Robot Syst Theory Appl. https://doi.org/10.1007/s10846-015-0178-2

    Article  Google Scholar 

  • Ashry S, Ogawa T, Gomaa W (2020) CHARM-deep: continuous human activity recognition model based on deep neural network using IMU sensors of smartwatch. IEEE Sens J 20(15):8757–8770. https://doi.org/10.1109/JSEN.2020.2985374

    Article  Google Scholar 

  • Assembly UG (1948) Universal declaration of human rights. UN General Assembly

  • Bacciu D, Di Rocco M, Dragone M, Gallicchio C, Micheli A, Saffiotti A (2019) An ambient intelligence approach for learning in smart robotic environments. Comput Intell. https://doi.org/10.1111/coin.12233

    Article  Google Scholar 

  • Bahuguna J, Weidel P, Morrison A (2018) Exploring the role of striatal D1 and D2 medium spiny neurons in action selection using a virtual robotic framework. Eur J Neurosci 49(6):737–753. https://doi.org/10.1111/ejn.14021

    Article  PubMed  PubMed Central  Google Scholar 

  • Bariselli S, Fobbs WC, Creed MC, Kravitz AV (2019) A competitive model for striatal action selection. Brain Res. https://doi.org/10.1016/j.brainres.2018.10.009

    Article  PubMed  Google Scholar 

  • Caine KE, Rogers WA, Fisk AD (2005) Privacy perceptions of an aware home with visual sensing devices. Proc Hum Factors Ergon Soc Annu Meet 49(21):1856–1858. https://doi.org/10.1177/154193120504902108

    Article  Google Scholar 

  • Calvaresi D, Cesarini D, Sernani P, Marinoni M, Dragoni AF, Sturm A (2017) Exploring the ambient assisted living domain: a systematic review. J Ambient Intell Humaniz Comput 8(2):239–257

    Article  Google Scholar 

  • Chen K, Zhang D, Yao L, Guo B, Yu Z, Liu Y (2021) Deep learning for sensor-based human activity recognition: overview, challenges, and opportunities. ACM Comput Surv (CSUR) 54(4):1–40

    Google Scholar 

  • Dawson TM, Golde TE, Lagier-Tourenne C (2018) Animal models of neurodegenerative diseases. Nat Neurosci 21(10):1370–1379. https://doi.org/10.1038/s41593-018-0236-8

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Drakopoulos F, Baby D, Verhulst S (2021) A convolutional neural-network framework for modelling auditory sensory cells and synapses. Commun Biol 4(1):1–17

    Article  Google Scholar 

  • Dura-Bernal S, Suter BA, Gleeson P, Cantarelli M, Quintana A, Rodriguez F, Kedziora DJ, Chadderdon GL, Kerr CC, Neymotin SA, McDougal RA, Hines M, Shepherd GM, Lytton WW (2019) NetPyNE, a tool for data-driven multiscale modeling of brain circuits. eLife 8:e44494. https://doi.org/10.7554/eLife.44494

    Article  PubMed  PubMed Central  Google Scholar 

  • Fernandes Junior FE, Yang G, Do HM, Sheng W (2016) Detection of privacy-sensitive situations for social robots in smart homes. In: Automation science and engineering (CASE), 2016 IEEE international conference on, pp 727–732. IEEE. https://doi.org/10.1109/COASE.2016.7743474

  • Garcia FA, Ranieri CM, Romero RAF (2019) Temporal approaches for human activity recognition using inertial sensors. In: Proceedings—2019 Latin American robotics symposium, 2019 Brazilian symposium on robotics and 2019 workshop on robotics in education, LARS/SBR/WRE 2019, pp 121–125. Institute of electrical and electronics engineers Inc. https://doi.org/10.1109/LARS-SBR-WRE48964.2019.00029

  • Georgievski I, Nguyen TA, Nizamic F, Setz B, Lazovik A, Aiello M (2017) Planning meets activity recognition: service coordination for intelligent buildings. Pervasive Mob Comput 38:110–139. https://doi.org/10.1016/j.pmcj.2017.02.008

    Article  Google Scholar 

  • Girard B, Tabareau N, Pham QC, Berthoz A, Slotine JJ (2008) Where neuroscience and dynamic system theory meet autonomous robotics: a contracting basal ganglia model for action selection. Neural Netw 21(4):628–641. https://doi.org/10.1016/j.neunet.2008.03.009

    Article  CAS  PubMed  Google Scholar 

  • Grisetti G, Stachniss C, Burgard W (2007) Improved techniques for grid mapping with Rao-Blackwellized particle filters. IEEE Trans Robot 23(1):34–46

    Article  Google Scholar 

  • Halje P, Brys I, Mariman JJ, Da Cunha C, Fuentes R, Petersson P (2019) Oscillations in cortico-basal ganglia circuits: implications for Parkinson’s disease and other neurologic and psychiatric conditions. J Neurophysiol 122(1):203–231. https://doi.org/10.1152/jn.00590.2018

    Article  PubMed  Google Scholar 

  • Haykin SS (2008) Neural networks: a comprehensive foundation, 3rd edn. Pearson, London

    Google Scholar 

  • Herath S, Harandi M, Porikli F (2017) Going deeper into action recognition: a survey. Image Vis Comput 60:4–21. https://doi.org/10.1016/j.imavis.2017.01.010

    Article  Google Scholar 

  • Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735

    Article  CAS  PubMed  Google Scholar 

  • Hwu TJ, Krichmar JL (2022) Neurorobotics: neuroscience and robots. In: Cangelosi A, Asada M (eds) Cognitive robotics, Chap 2. MIT Press, Cambridge, pp 19–40

    Chapter  Google Scholar 

  • Imran J, Raman B (2020) Evaluating fusion of RGB-D and inertial sensors for multimodal human action recognition. J Ambient Intell Humaniz Comput 11(1):189–208. https://doi.org/10.1007/s12652-019-01239-9

    Article  Google Scholar 

  • Kita H, Kita T (2011) Cortical stimulation evokes abnormal responses in the dopamine-depleted rat basal ganglia. J Neurosci 31(28):10311–10322. https://doi.org/10.1523/JNEUROSCI.0915-11.2011

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Koenig N, Howard A (2004) Design and use paradigms for gazebo, an open-source multi-robot simulator. In: 2004 IEEE/RSJ international conference on intelligent robots and systems (IROS)(IEEE Cat. No. 04CH37566), vol 3, pp 2149–2154. IEEE

  • Könings B, Schaub F, Weber M (2016) Privacy and trust in ambient intelligent environments. In: Next generation intelligent environments. Springer, pp 133–164

  • Koprich JB, Kalia LV, Brotchie JM (2017) Animal models of alpha-synucleinopathy for Parkinson disease drug development. Nat Rev Neurosci 18(9):515–529. https://doi.org/10.1038/nrn.2017.75

    Article  CAS  PubMed  Google Scholar 

  • Krichmar JL (2018) Neurorobotics-a thriving community and a promising pathway toward intelligent cognitive robots. Front Neurorobotics 12:42

    Article  Google Scholar 

  • Kumaravelu K, Brocker DT, Grill WM (2016) A biophysical model of the cortex-basal ganglia-thalamus network in the 6-OHDA lesioned rat model of Parkinson’s disease. J Comput Neurosci 40(2):207–229. https://doi.org/10.1007/s10827-016-0593-9

    Article  PubMed  PubMed Central  Google Scholar 

  • Lakshmanan V, Robinson S, Munn M (2020) Machine learning design patterns. O’Reilly Media, Sebastopol

    Google Scholar 

  • Lánský P, Rodriguez R, Sacerdote L (2004) Mean instantaneous firing frequency is always higher than the firing rate. Neural Comput 16(3):477–489. https://doi.org/10.1162/089976604772744875

    Article  PubMed  Google Scholar 

  • Li J, Li Z, Chen F, Bicchi A, Sun Y, Fukuda T (2019) Combined sensing, cognition, learning, and control for developing future neuro-robotics systems: a survey. IEEE Trans Cognit Dev Syst 11(2):148–161. https://doi.org/10.1109/TCDS.2019.2897618

    Article  Google Scholar 

  • Li Z, Liu F, Yang W, Peng S, Zhou J (2021) A survey of convolutional neural networks: analysis, applications, and prospects. IEEE Trans Neural Netw Learn Syst. https://doi.org/10.1109/TNNLS.2021.3084827

    Article  PubMed  PubMed Central  Google Scholar 

  • Liang Y, Yan Z, Zhang Q, Liang H, Ji X, Liu Y, Liu R (2019) A decision-making model based on basal ganglia account of action prediction. In: IEEE international conference on robotics and biomimetics, ROBIO 2019. Institute of electrical and electronics engineers Inc., pp 1705–1710 https://doi.org/10.1109/ROBIO49542.2019.8961538

  • Liénard J, Girard B (2014) A biologically constrained model of the whole basal ganglia addressing the paradoxes of connections and selection. J Comput Neurosci 36(3):445–468. https://doi.org/10.1007/s10827-013-0476-2

    Article  PubMed  Google Scholar 

  • Li K, Wu J, Zhao X, Tan M (2019) Real-time human-robot interaction for a service robot based on 3D human activity recognition and human-mimicking decision mechanism. In: 8th annual IEEE international conference on cyber technology in automation, control and intelligent systems, CYBER 2018, pp 498–503. Institute of electrical and electronics engineers Inc. https://doi.org/10.1109/CYBER.2018.8688272

  • Lu Y, Velipasalar S (2019) Autonomous human activity classification from wearable multi-modal sensors. IEEE Sens J 19(23):11403–11412. https://doi.org/10.1109/JSEN.2019.2934678

    Article  Google Scholar 

  • Luu DK, Nguyen AT, Jiang M, Xu J, Drealan MW, Cheng J, Keefer EW, Zhao Q, Yang Z (2021) Deep learning-based approaches for decoding motor intent from peripheral nerve signals. Front Neurosci 15:667907

    Article  PubMed  PubMed Central  Google Scholar 

  • Ma CY, Chen MH, Kira Z, AlRegib G (2019) TS-LSTM and temporal-inception: Exploiting spatiotemporal dynamics for activity recognition. Signal Processing: Image Communication 71:76–87. https://doi.org/10.1016/j.image.2018.09.003

    Article  Google Scholar 

  • Markowitz JE, Gillis WF, Beron CC, Neufeld SQ, Robertson K, Bhagat ND, Peterson RE, Peterson E, Hyun M, Linderman SW, Sabatini BL, Datta SR (2018) The striatum organizes 3D behavior via moment-to-moment action selection. Cell 174(1):44–58. https://doi.org/10.1016/j.cell.2018.04.019

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • McGregor MM, Nelson AB (2019) Circuit mechanisms of Parkinson’s disease. Neuron 101(6):1042–1056. https://doi.org/10.1016/j.neuron.2019.03.004

    Article  CAS  PubMed  Google Scholar 

  • Merk T, Peterson V, Köhler R, Haufe S, Richardson RM, Neumann WJ (2022) Machine learning based brain signal decoding for intelligent adaptive deep brain stimulation. Exp Neurol. https://doi.org/10.1016/j.expneurol.2022.113993

    Article  PubMed  Google Scholar 

  • Mojarad R, Attal F, Chibani A, Fiorini SR, Amirat Y (2018) Hybrid approach for human activity recognition by ubiquitous robots. In: IEEE international conference on intelligent robots and systems. Institute of electrical and electronics engineers Inc., pp 5660–5665 https://doi.org/10.1109/IROS.2018.8594173

  • Mora-Sánchez A, Dreyfus G, Vialatte FB (2019) Scale-free behaviour and metastable brain-state switching driven by human cognition, an empirical approach. Cogn Neurodyn 13(5):437–452

    Article  PubMed  PubMed Central  Google Scholar 

  • Mulcahy G, Atwood B, Kuznetsov A (2020) Basal ganglia role in learning rewarded actions and executing previously learned choices: healthy and diseased states. PLoS ONE 15(2):1–26

    Article  Google Scholar 

  • Oh SL, Hagiwara Y, Raghavendra U, Yuvaraj R, Arunkumar N, Murugappan M, Acharya UR (2018) A deep learning approach for Parkinson’s disease diagnosis from EEG signals. Neural Comput Appl. https://doi.org/10.1007/s00521-018-3689-5

    Article  Google Scholar 

  • Ordóñez F, Roggen D (2016) Deep convolutional and LSTM recurrent neural networks for multimodal wearable activity recognition. Sensors 16(1):115. https://doi.org/10.3390/s16010115

    Article  PubMed  PubMed Central  Google Scholar 

  • Pimentel JM, Moioli RC, De Araujo MF, Ranieri CM, Romero RA, Broz F, Vargas PA (2021) Neuro4pd: an initial neurorobotics model of Parkinson’s disease. Front Neurorobotics 88

  • Prescott TJ, Montes González FM, Gurney K, Humphries MD, Redgrave P (2006) A robot model of the basal ganglia: behavior and intrinsic processing. Neural Netw 19(1):31–61. https://doi.org/10.1016/j.neunet.2005.06.049

    Article  PubMed  Google Scholar 

  • Pronin S, Wellacott L, Pimentel J, Moioli RC, Vargas PA (2021) Neurorobotic models of neurological disorders: a mini review. Front Neurorobotics 15:26

    Article  Google Scholar 

  • Quigley M, Conley K, Gerkey B, Faust J, Foote T, Leibs J, Wheeler R, Ng AY (2009) ROS: an open-source robot operating system. In: ICRA workshop on open source software, vol 3, p 5. Kobe, Japan

  • Ranieri CM, MacLeod S, Dragone M, Vargas PA, Romero RAF (2021) Human activities with videos, inertial units and ambient sensors. Dryad Digital Repository

  • Ranieri CM, Nardari G, Pinto AH, Tozadore DC, Romero RA (2018) LARa: a robotic framework for human-robot interaction on indoor environments. In: Proceedings—15th Latin American robotics symposium, 6th Brazilian robotics symposium and 9th workshop on robotics in education, LARS/SBR/WRE 2018. Institute of electrical and electronics engineers Inc., pp 383–389 https://doi.org/10.1109/LARS/SBR/WRE.2018.00074

  • Ranieri CM, MacLeod S, Dragone M, Vargas PA, Romero RF (2021) Activity recognition for ambient assisted living with videos, inertial units and ambient sensors. Sensors 21(3):768. https://doi.org/10.3390/S21030768

    Article  PubMed  PubMed Central  Google Scholar 

  • Ranieri CM, Pimentel JM, Romano MR, Elias LA, Romero RA, Lones MA, Araujo MF, Vargas PA, Moioli RC (2021) A data-driven biophysical computational model of Parkinson’s disease based on marmoset monkeys. IEEE Access 9:122548–122567

    Article  Google Scholar 

  • Ranieri C, Moioli R, Romero R, De Araujo M, De Santana M, Pimentel J, Vargas P (2020) Unveiling Parkinson’s disease features from a primate model with deep neural networks. In: Proceedings of the international joint conference on neural networks. https://doi.org/10.1109/IJCNN48605.2020.9207180

  • Ranieri C, Vargas P, Romero R (2020) Uncovering human multimodal activity recognition with a deep learning approach. In: Proceedings of the international joint conference on neural networks. https://doi.org/10.1109/IJCNN48605.2020.9207255

  • Rodriguez Lera FJ, Martín Rico F, Guerrero Higueras AM, Olivera VM (2020) A context-awareness model for activity recognition in robot-assisted scenarios. Expert Syst 37(2):e12481. https://doi.org/10.1111/exsy.12481

    Article  Google Scholar 

  • Rodríguez-Moreno I, Martínez-Otzeta JM, Sierra B, Rodriguez I, Jauregi E (2019) Video activity recognition: state-of-the-art. Sensors 19(14):3160. https://doi.org/10.3390/s19143160

    Article  PubMed  PubMed Central  Google Scholar 

  • Rucci M, Bullock D, Santini F (2007) Integrating robotics and neuroscience: brains for robots, bodies for brains. Adv Robot 21(10):1115–1129

    Article  Google Scholar 

  • Saeidi M, Karwowski W, Farahani FV, Fiok K, Taiar R, Hancock P, Al-Juaid A (2021) Neural decoding of EEG signals with machine learning: a systematic review. Brain Sci 11(11):1525

    Article  PubMed  PubMed Central  Google Scholar 

  • Selvaggio M, Cognetti M, Nikolaidis S, Ivaldi S, Siciliano B (2021) Autonomy in physical human-robot interaction: a brief survey. IEEE Robot Autom Lett. https://doi.org/10.1109/LRA.2021.3100603

    Article  Google Scholar 

  • Sen S, Daimi SN, Watanabe K, Takahashi K, Bhattacharya J, Saha G (2020) Switch or stay? automatic classification of internal mental states in bistable perception. Cogn Neurodyn 14(1):95–113

    Article  PubMed  Google Scholar 

  • Sharkey A, Sharkey N (2012) Granny and the robots: ethical issues in robot care for the elderly. Ethics Inf Technol 14(1):27–40. https://doi.org/10.1007/s10676-010-9234-6

    Article  Google Scholar 

  • Steven Eyobu O, Han D (2018) Feature representation and data augmentation for human activity classification based on wearable IMU sensor data using a deep LSTM neural network. Sensors 18(9):2892. https://doi.org/10.3390/s18092892

    Article  PubMed  PubMed Central  Google Scholar 

  • Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z (2016) Rethinking the inception architecture for computer vision. In: IEEE conference on computer vision and pattern recognition (CVPR). IEEE, Las Vegas, NV, USA, pp 2818–2826

  • Tinkhauser G, Pogosyan A, Tan H, Herz DM, Kühn AA, Brown P (2017) Beta burst dynamics in Parkinson’s disease off and on dopaminergic medication. Brain 140(11):2968–2981. https://doi.org/10.1093/brain/awx252

    Article  PubMed  Google Scholar 

  • van Albada SJ, Robinson PA (2009) Mean-field modeling of the basal ganglia-thalamocortical system. I. Firing rates in healthy and parkinsonian states. J Theor Biol 257(4):642–663. https://doi.org/10.1016/j.jtbi.2008.12.018

    Article  PubMed  Google Scholar 

  • van der Heijden K, Mehrkanoon S (2022) Goal-driven, neurobiological-inspired convolutional neural network models of human spatial hearing. Neurocomputing 470:432–442

    Article  Google Scholar 

  • Van Der Smagt P, Arbib MA, Metta G (2016) Neurorobotics: from vision to action. In: Springer handbook of robotics. Springer International Publishing, pp 2069–2094

  • Wojtowytsch S, Weinan E (2020) Can shallow neural networks beat the curse of dimensionality? A mean field training perspective. IEEE Trans Artif Intell 1(2):121–129

    Article  Google Scholar 

  • Zach C, Pock T, Bischof H (2007) A duality based approach for real-time TV-L 1 optical flow. In: Joint pattern recognition symposium. Springer, pp 214–223

  • Zahra O, Navarro-Alarcon D, Tolu S (2021) A neurorobotic embodiment for exploring the dynamical interactions of a spiking cerebellar model and a robot arm during vision-based manipulation tasks. Int J Neural Syst 32:2150028

    Article  PubMed  Google Scholar 

  • Zeiler MD, Fergus R (2014) Visualizing and understanding convolutional networks. In: European conference on computer vision. Springer, pp 818–833

  • Zheng P, Kozloski J (2017) Striatal network models of Huntington’s disease dysfunction phenotypes. Front Comput Neurosci 11:70

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Funding

This work was funded by the Sao Paulo Research Foundation (FAPESP), grants 2017/02377-5, 2017/01687-0, 2018/25902-0 and 2021/10921-2, and the Neuro4PD project - Royal Society and Newton Fund (NAF/ R2/180773). Moioli acknowledge the support from the Brazilian institutions: INCT INCEMAQ of the CNPq/MCTI, FAPERN, CAPES, FINEP, and MEC. This research was carried out using the computational resources from the CeMEAI funded by FAPESP, grant 2013/07375-0. Additional resources were provided by the Robotics Lab within the ECR, and by the Nvidia Grants program.

Author information

Authors and Affiliations

Authors

Contributions

All authors contributed to the design of the experiments. Ranieri provided the specific methods and implementations, performed the experiments and analysed the results. Moioli, Vargas and Romero revised the methods and results presented, contributing to the discussion. Ranieri written the draft of the paper, revised by the other authors.

Corresponding author

Correspondence to Caetano M. Ranieri.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Consent for publication

All authors agreed to publish this paper in its present form.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ranieri, C.M., Moioli, R.C., Vargas, P.A. et al. A neurorobotics approach to behaviour selection based on human activity recognition. Cogn Neurodyn 17, 1009–1028 (2023). https://doi.org/10.1007/s11571-022-09886-z

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11571-022-09886-z

Keywords

Navigation