Skip to main content
SpringerLink
Log in

Development of a humanoid robot control system based on AR-BCI and SLAM navigation

  • Research Article
  • Published:
Cognitive Neurodynamics Aims and scope Submit manuscript

Abstract

Brain-computer interface (BCI)-based robot combines BCI and robotics technology to realize the brain’s intention to control the robot, which not only opens up a new way for the daily care of the disabled individuals, but also provides a new way of communication for normal people. However, the existing systems still have shortcomings in many aspects such as friendliness of human–computer interaction, and interaction efficient. This study developed a humanoid robot control system by integrating an augmented reality (AR)-based BCI with a simultaneous localization and mapping (SLAM)-based scheme for autonomous indoor navigation. An 8-target steady-state visual evoked potential (SSVEP)-based BCI was implemented to enable direct control of the humanoid robot by the user. A Microsoft HoloLens was utilized to display visual stimuli for eliciting SSVEPs. Filter bank canonical correlation analysis (FBCCA), a training-free method, was used to detect SSVEPs in this study. By leveraging SLAM technology, the proposed system alleviates the need for frequent control commands transmission from the user, thereby effectively reducing their workload. Online results from 12 healthy subjects showed this developed BCI system was able to select a command out of eight potential targets with an average accuracy of 94.79%. The autonomous navigation subsystem enabled the humanoid robot to autonomously navigate to a destination chosen utilizing the proposed BCI. Furthermore, all participants successfully completed the experimental task using the developed system without any prior training. These findings illustrate the feasibility of the developed system and its potential to contribute novel insights into humanoid robots control strategies.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Data availability

Data will be made available on request.

References

  • Abdulsaheb J, Kadhim D (2023) Real-time SLAM mobile robot and navigation based on cloud-based implementation. J Robot 2023:9967236

    Google Scholar 

  • Ayaka F, Kristiina J, Kei O et al (2022) Development of dialogue system architecture toward co-creating social intelligence when talking with a partner robot. Front Robot AI 9:933001

    Article  Google Scholar 

  • Baniqued P, Stanyer E, Awais M et al (2021) Brain-computer interface robotics for hand rehabilitation after stroke: a systematic review. J NeuroEng Rehabil 18(1):15

    Article  PubMed  PubMed Central  Google Scholar 

  • Barbehenn M (1998) A note on the complexity of Dijkstra’s algorithm for graphs with weighted vertices. IEEE Trans Comput 47(2):263

    Article  Google Scholar 

  • Bell C, Shenoy P, Chalodhorn R et al (2008) Control of a humanoid robot by a noninvasive brain-computer interface in humans. J Neural Eng 5(2):214–220

    Article  PubMed  Google Scholar 

  • Berg J, Lu S (2020) Review of interfaces for industrial human-robot interaction. Cur Robot Rep 1:27–34

    Article  Google Scholar 

  • Cadena C, Carlone L, Carrillo H et al (2016) Past, present, and future of simultaneous localization and mapping: toward the robust-perception age. IEEE Trans Robot 32(6):1309–1332

    Article  Google Scholar 

  • Canzobre D, Regueiro C, Calvo-Varela L et al (2016) Integration of 3-D perception and autonomous computation on a NAO humanoid robot. In: Robot 2015: second Iberian robotics conference, 1:161−173

  • Chae Y, Jeong J, Jo S (2012) Toward brain-actuated humanoid robots: asynchronous direct control using an EEG-based BCI. IEEE Trans Robot 28(5):1131–1144

    Article  Google Scholar 

  • Chamola V, Vineet A, Nayyar A et al (2020) Brain-computer interface-based humanoid control: a review. Sensors 20(13):3620

    Article  PubMed  PubMed Central  Google Scholar 

  • Chella A, Pagello E, Menegatti E et al (2009) A BCI teleoperated museum robotic guide. In: Proceedings of the 2009 international conference on complex, 16:783–788

  • Chen X, Wang Y, Nakanishi M et al (2015) High-speed spelling with a noninvasive brain–computer interface. Proc Natl Acad Sci 112(44):E6058–E6067

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Chen X, Wang Y, Gao S et al (2015) Filter bank canonical correlation analysis for implementing a high-speed SSVEP-based brain-computer interface. J Neural Eng 12(4):14

    Article  Google Scholar 

  • Chen X, Zhao B, Wang Y et al (2018) Control of a 7-DOF robotic arm system with an SSVEP-based BCI. Int J Neural Syst 28(8):1850018

    Article  PubMed  Google Scholar 

  • Chen X, Zhao B, Wang Y et al (2019) Combination of high-frequency SSVEP-based BCI and computer vision for controlling a robotic arm. J Neural Eng 16(2):026012

    Article  PubMed  Google Scholar 

  • Chen X, Huang X, Wang Y et al (2020) Combination of augmented reality based brain-computer interface and computer vision for high-level control of a robotic arm. IEEE Trans Neural Syst Rehabil Eng 28(12):3140–3147

    Article  PubMed  Google Scholar 

  • Chen L, Chen P, Zhao S et al (2022) Adaptive asynchronous control system of robotic arm based on augmented reality-assisted brain–computer interface. J Neural Eng 18(9):066005

  • Chikurtev D (2015) Optimizing the navigation for mobile robot for inspection by using robot operating system. Prob Eng Cybern Robot 66:63–74

    Google Scholar 

  • Chong T, Tang X, Leng C et al (2015) Sensor technologies and simultaneous localization and mapping (SLAM). Procedia Comput Sci 76:174–179

    Article  Google Scholar 

  • Evripidou S, Georgiou K, Doitsidis L et al (2020) Educational robotics: platforms, competitions and expected learning outcomes. IEEE Access 8:219534–219562

    Article  Google Scholar 

  • Flesher S, Downey J, Weiss J et al (2021) A brain-computer interface that evokes tactile sensations improves robotic arm control. Science 372(6544):831–836

  • Fu R, Feng X, Wang S et al (2024) Control of the robotic arm system with an SSVEP-based BCI. Meas Sci Technol 35(5):055702

    Article  Google Scholar 

  • Gao X, Wang Y, Chen X et al (2021) Interface, interaction, and intelligence in generalized brain-computer interfaces. Trends Cogn Sci 25(8):671–684

    Article  PubMed  Google Scholar 

  • Gao S, Wang Y, Gao X et al (2014) Visual and auditory brain-computer interfaces. IEEE Trans Biomed Eng 61(5):1436–1447

  • Gatesichapakorn S, Takamatsu J, Ruchanurucks M (2019) ROS based autonomous mobile robot navigation using 2D LiDAR and RGB-D camera. Proceedings of 1st international symposium on instrumentation, control, artificial intelligence, and robotics (ICA-SYMP). pp 151–154

  • Han X, Lin K, Gao S et al (2019) A novel system of SSVEP-based human-robot coordination. J Neural Eng 16(1):016006

    Article  PubMed  Google Scholar 

  • Henrik A, Jonas L, Stephanie L (2022) A local planner for accurate positioning for a multiple steer-and-drive unit vehicle using non-linear optimization. Sensors 22(7):2588

    Article  Google Scholar 

  • Hochberg L, Bacher D, Jarosiewicz B et al (2012) Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature 485(7398):372–375

  • Kam H, Lee S, Park T et al (2015) RViz: a toolkit for real domain data visualization. Telecommun Syst 60:337–345

    Article  Google Scholar 

  • Ke Y, Liu P, An X et al (2020) An online SSVEP-BCI system in an optical see-through augmented reality environment. J Neural Eng 17(1):016066

    Article  PubMed  Google Scholar 

  • Khademi Z, Ebrahimi F, Kordy H (2023) A review of critical challenges in MI-BCI: from conventional to deep learning methods. J Neurosci Methods 383:109736

    Article  PubMed  Google Scholar 

  • Labbé M, Michaud F (2019) RTAB-Map as an open-source lidar and visual simultaneous localization and mapping library for large-scale and long-term online operation. J Field Robot 36(2):416–446

    Article  Google Scholar 

  • Lee Y, Hwang S, Park J (2015) Balancing of humanoid robot using contact force/moment control by task-oriented whole body control framework. Auton Robots 40:457–472

    Article  Google Scholar 

  • Li W, Li M, Zhao J (2015) Control of humanoid robot via motion-onset visual evoked potentials. Front Syst Neurosci 9(8):247

    Google Scholar 

  • Li M, Chen X, Cui H (2023) A high-frequency SSVEP-BCI system based on simultaneous modulation of luminance and motion using intermodulation frequencies. IEEE Trans Neural Syst Rehabil Eng 31:2603–2611

  • Mao X, Li W, Lei C et al (2019) A brain-robot interaction system by fusing human and machine intelligence. IEEE Trans Neural Syst Rehabil Eng 27(3):533–542

    Article  PubMed  Google Scholar 

  • McFarland D, Wolpaw J (2011) Brain-computer interfaces for communication and control. Clin Neurophysiol 113(6):767–791

    Google Scholar 

  • Melkas H, Hennala L, Pekkarinen S et al (2020) Impacts of robot implementation on care personnel and clients in elderly-care institutions. Int J Med Inform 134:104041

    Article  PubMed  Google Scholar 

  • Petit D, Gergondet P, Cherubini A et al (2014) A. Navigation assistance for a BCI-controlled humanoid robot. In: Proceedings of the 4th annual IEEE international conference on cyber technology in automation, 4:246–251

  • Qidwai U, Kashem SB, Conor O (2020) Humanoid robot as a teacher’s assistant: helping children with autism to learn social and academic skills. J Intell Robot Syst 98(3–4):759–770

    Article  Google Scholar 

  • Saduanov B, Alizadeh T, An J et al (2018) Trained by demonstration humanoid robot controlled via a BCI system for telepresence. In: Proceedings of the 2018 6th international conference on brain-computer interface (BCI), 15:1–4

  • Santiago F, Ruben H, Antonio A et al (2022) How do consumers’ gender and rational thinking affect the acceptance of entertainment social robots? Int J Soc Robot 14(4):973–994

    Article  Google Scholar 

  • Saunderson S, Nejat G (2019) How robots influence humans: a survey of nonverbal communication in social human-robot interaction. Int J Soc Robot 11:575–608

    Article  Google Scholar 

  • Shan T, Englot B, Ratti C et al (2021) Lvi-sam: tightly-coupled lidar-visual-inertial odometry via smoothing and mapping. In: 2021 IEEE international conference on robotics and automation (ICRA). pp 5692–5698

  • Singla R, Khosla A, Jha R (2013) Influence of stimuli color on steady-state visual evoked potentials based BCI wheelchair control. J Biomed Sci Eng 6(11):1050

    Article  Google Scholar 

  • Spataro R, Chella A, Allison B et al (2017) Reaching and grasping a glass of water by locked-in ALS patients through a BCI-controlled humanoid robot. Front Hum Neurosci 1(11):68

    Google Scholar 

  • Tonin L, Millan J (2021) Noninvasive brain-machine interfaces for robotic devices. Annu Rev Control Robot Auton Syst 4:191–214

  • Vidya N, Bryan G, Kevin M et al (2022) A brain-computer interface for teleoperation of a semiautonomous mobile robotic assistive system using SLAM. J Robot 2022:6178917

    Google Scholar 

  • Wang M, Li R, Zhang R et al (2018) A wearable SSVEP-based BCI system for quadcopter control using head-mounted device. IEEE Access 6:26789–26798

    Article  Google Scholar 

  • Xu B, Liu D, Xue M et al (2023) Continuous shared control of a mobile robot with brain-computer interface and autonomous navigation for daily assistance. Comp Struct Biotechnol J 22:3–16

    Article  CAS  Google Scholar 

  • Yuan F, Klavon E, Liu Z et al (2021) A systematic review of robotic rehabilitation for cognitive training. Front Robot AI 11(8):605715

  • Zhang S, Gao X, Chen X (2022) Humanoid robot walking in maze controlled by SSVEP-BCI based on augmented reality stimulus. Front Hum Neurosci 16:908050

    Article  PubMed  PubMed Central  Google Scholar 

  • Zhang J, Gao S, Zhou K et al (2023) An online hybrid BCI combining SSVEP and EOG-based eye movements. Front Hum Neurosci 17:1103935

    Article  PubMed  PubMed Central  Google Scholar 

  • Zhang S, Chen Y, Zhang L et al (2023) Study on robot grasping system of SSVEP-BCI based on augmented reality stimulus. Tsinghua Sci Technol 28(2):322–329

    Article  Google Scholar 

  • Zhao J, Li W, Li M (2015) Comparative study of SSVEP- and P300-based models for the telepresence control of humanoid robots. PLoS ONE 10(11):e0142168

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Funding

National Key R & D Program of China, 2022YFC3602803, Xiaogang Chen, National Natural Science Foundation of China, 62171473, Xiaogang Chen, Tianjin Municipal Science and Technology Plan Project, 21JCYBJC01500, Xiaogang Chen, Fundamental Research Funds for the Central Universities (No. 3332023170), Meng Li.

Author information

Authors and Affiliations

  1. Department of Life Sciences, Tiangong University, Tianjin, 300387, China

    Yao Wang & Mingxing Zhang

  2. Institute of Biomedical Engineering, Chinese Academy of Medical Sciences and Peking Union Medical College, Tianjin, 300192, China

    Meng Li, Hongyan Cui & Xiaogang Chen

Authors
  1. Yao Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mingxing Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Meng Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hongyan Cui
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Xiaogang Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xiaogang Chen.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

This work was supported in part by National Key R&D Program of China (No. 2022YFC3602803), National Natural Science Foundation of China (No. 62171473), Tianjin Municipal Science and Technology Plan Project (No. 21JCYBJC01500), Fundamental Research Funds for the Central Universities (No. 3332023170). (Corresponding author: Xiaogang Chen).

This work involved human subjects or animals in its research. Approval of all ethical and experimental procedures and protocols was granted by the Institutional Review Board of Tsinghua University.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wang, Y., Zhang, M., Li, M. et al. Development of a humanoid robot control system based on AR-BCI and SLAM navigation. Cogn Neurodyn (2024). https://doi.org/10.1007/s11571-024-10122-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s11571-024-10122-z

Keywords

Search

Navigation