Human-Robot Cooperation via Brain Computer Interface in Assistive Scenario

  • G. Foresi
  • A. Freddi
  • S. IarloriEmail author
  • S. Longhi
  • A. Monteriù
  • D. Ortenzi
  • D. Proietti Pagnotta
Conference paper
Part of the Lecture Notes in Electrical Engineering book series (LNEE, volume 540)


In the last years, the development of robots for assisting and collaborating with people has experienced a large growth. Applications for assistive robots include hospital service robots, factory intelligent assistants and personal homecare robots. Working in shared environments with human beings, these robots require effective ways to achieve an increasing human-robot cooperation. This work presents a possible approach for performing human-robot cooperation, namely recognition of a user selected object by means of Brain Computer Interface (BCI), followed by pick and place via a robotic arm. The object selection is achieved introducing a BCI that allows the user, after a training phase, to choose one among six different objects of common diffusion. The selection is achieved by interpreting the P300 signals generated in the brain, when the image of the object, desidered by the user, appears on a computer screen as a visual stimulus. The robot then recognizes, through a classifier, the selected object among others within its workspace, and inscribes it in a rectangle shape. Finally, the robot arm is moved in correspondence to the object position, the gripper is rotated according to the object orientation and the object grasped and moved into a different position on a desk in front of the robot. This system could support people with limited motor skills or paralysis, playing an important role in structured assistive environments in a near future.



Authors would like to thank Simone Hanisch, Davide Centioni, Massimo Martini and Diego Retaggi, who contributed to some of the algorithms described in the paper.


  1. 1.
    Diamantini, C., Freddi, A., Longhi, S., Potena, D., & Storti, E. (2016, July). A goal-oriented, ontology-based methodology to support the design of AAL environments. Expert Systems with Applications, 64, 117–131.CrossRefGoogle Scholar
  2. 2.
    Longhi, S., Monteriù, A., & Freddi, A. (Eds.). (2017). Human monitoring, smart health and assisted living: techniques and technologies. IET.Google Scholar
  3. 3.
    Benetazzo, F., Freddi, A., Monteriù, A., & Longhi, S. (2014, September). Respiratory rate detection algorithm based on RGB-D camera: Theoretical background and experimental results. Healthcare Technology Letters, 1(3), 81–86.CrossRefGoogle Scholar
  4. 4.
    Freddi, A., Longhi, S., Monteriù, A., & Ortenzi, D. (2016, September). Redundancy analysis of cooperative dual-arm manipulators. International Journal of Advanced Robotic Systems, 13(5).CrossRefGoogle Scholar
  5. 5.
    Benetazzo, F., Ferracuti, F., Freddi, A., Giantomassi, A., Iarlori, S., Longhi, S., et al. (2015, July) AAL technologies for independent life of elderly people. In Biosystems & biorobotics, ambient assisted living: Italian forum 2014 (Vol. 11, pp. 329–343). Springer International Publishing.Google Scholar
  6. 6.
    Cavallo, F., Marletta, V., Monteriù, A., & Siciliano, P. (Eds.). (2017). Ambient assisted living: Italian forum 2016. Springer.Google Scholar
  7. 7.
    Schröer, S., Killmann, I., Frank, B., Völker, M., Fiederer, L., Ball, T., et al. (2015, May) An autonomous robotic assistant for drinking. In 2015 IEEE International Conference on Robotics and Automation (ICRA) (pp. 6482–6487).Google Scholar
  8. 8.
    Edsinger, A., & Kemp, C. C. (2007, August) Human-robot interaction for cooperative manipulation: Handing objects to one another. In RO-MAN 2007—The 16th IEEE International Symposium on Robot and Human Interactive Communication (pp. 1167–1172).Google Scholar
  9. 9.
    Wang, H., Ma, H., Yang, C., & Yang, Y. (2015, December) Simulation of one effective human-robot cooperation method based on kinect sensor and uncalibrated visual servoing. In 2015 IEEE International Conference on Robotics and Biomimetics (ROBIO) (pp. 2014–2019).Google Scholar
  10. 10.
    Donner, P., Mörtl, A., Hirche, S., & Buss, M. (2013, May) Human-robot cooperative object swinging. In 2013 IEEE International Conference on Robotics and Automation (pp. 4343–4349).Google Scholar
  11. 11.
    Awano, H., Ogata, T., Nishide, S., Takahashi, T., Komatani, K., & Okuno, H.G. (2010, October) Human-robot cooperation in arrangement of objects using confidence measure of neuro-dynamical system. In 2010 IEEE International Conference on Systems, Man and Cybernetics (pp. 2533–2538).Google Scholar
  12. 12.
    Gomez, R., Nakamura, K., Mizumoto, T., & Nakadai, K. (2015, November) Compensating changes in speaker position for improved voice-based human-robot communication. In 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids) (pp. 977–982).Google Scholar
  13. 13.
    Strobel, M., Illmann, J., Kluge, B., & Marrone, F. (2002). Using spatial context knowledge in gesture recognition for commanding a domestic service robot. In Proceedings. 11th IEEE International Workshop on Robot and Human Interactive Communication (pp. 468–473).Google Scholar
  14. 14.
    Rogalla, O., Ehrenmann, M., Zollner, R., Becher, R., & Dillmann, R. (2002). Using gesture and speech control for commanding a robot assistant. In Proceedings. 11th IEEE International Workshop on Robot and Human Interactive Communication (pp. 454–459).Google Scholar
  15. 15.
    Hochberg, L., Bacher, D., Jarosiewicz, B., Masse, N. Y., Simeral, J. D., Vogel, J., et al. (2012). Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature, 372–275.CrossRefGoogle Scholar
  16. 16.
    Eleni, A. (2008, October). Control of medical robotics and neurorobotic prosthetics by noninvasive brain-robot interfaces via EEG and RFID technology. In 2008 8th IEEE International Conference on BioInformatics and BioEngineering (pp. 1–4).Google Scholar
  17. 17.
    Minati, L., Yoshimura, N., & Koike, Y. (2016). Hybrid control of a vision-guided robot arm by EOG, EMG, EEG biosignals and head movement acquired via a consumer-grade wearable device. IEEE Access, 4, 9528–9541.CrossRefGoogle Scholar
  18. 18.
    Ferracuti, F., Freddi, A., Iarlori, S., Longhi, S., & Peretti, P. (2013, November). Auditory paradigm for a P300 BCI system using spatial hearing. In 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 871–876).Google Scholar
  19. 19.
    McFarland, D. J., & Wolpaw, J. R. (2011, May). Brain-computer interfaces for communication and control. Communications of the ACM, 54(5), 60–66.CrossRefGoogle Scholar
  20. 20.
    Sorbello, R., Tramonte, S., Giardina, M., Bella, V. L., Spataro, R., Allison, B., et al. (2017). A human-humanoid interaction through the use of BCI for locked-in ALS patients using neuro-biological feedback fusion. IEEE Transactions on Neural Systems and Rehabilitation Engineering, PP(99), 1–1.Google Scholar
  21. 21.
    Petit, D., Gergondet, P., Cherubini, A., Meilland, M., Comport, A.I., & Kheddar, A. (2014, June) Navigation assistance for a BCI-controlled humanoid robot. In The 4th Annual IEEE International Conference on Cyber Technology in Automation, Control and Intelligent (pp. 246–251).Google Scholar
  22. 22.
    Tang, J., Jiang, J., Yu, Y., & Zhou, Z. (2015, November) Humanoid robot operation by a brain-computer interface. In 2015 7th International Conference on Information Technology in Medicine and Education (ITME) (pp. 476–479).Google Scholar
  23. 23.
    Wijayasekara, D., & Manic, M. (2013, June) Human machine interaction via brain activity monitoring. In 2013 6th International Conference on Human System Interactions (HSI) (pp. 103–109).Google Scholar
  24. 24.
    Salazar-Gomez, A. F., DelPreto, J., Gil, S., Guenther, F., & Rus, D. (2017). Correcting robot mistakes in real time using EEG signals. In 2017 IEEE International Conference on Robotics and Automation (ICRA).Google Scholar
  25. 25.
    Jia, W., Huang, D., Bai, O., Pu, H., Luo, X., & Chen, X. (2012). Reliable planning and execution of a human-robot cooperative system based on noninvasive brain-computer interface with uncertainty. In IEEE International Conference on Intelligent Robots and Systems (10).Google Scholar
  26. 26.
    Foresi, G., Freddi, A., Kyrki, V., Monteriù, A., Muthusamy, R., Ortenzi, D., et al. (2017). An avoidance control strategy for joint-position limits of dual-arm robots. In Proceedings of the 20th IFAC World Congress (to be published into).CrossRefGoogle Scholar
  27. 27.
    Viola, P., & Jones, M. (2001). Rapid object detection using a boosted cascade of simple features. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Vol. 1, pp. I–511–I–518).Google Scholar
  28. 28.
    Ferreira, A., Celeste, W., auat cheein, F., Bastos, T., Sarcinelli-Filho, M., & Carelli, R. (2008). Human-machine interfaces based on EMG and EEG applied to robotic systems 5(02), 10.Google Scholar
  29. 29.
    Mao, X., Li, M., Li, W., Niu, L., Xian, B., Zeng, M., et al. (2017). Review article progress in EEG-based brain robot interaction systems 2017(04), 1–25.Google Scholar
  30. 30.
    Kim, Y. J., Park, S. W., Yeom, H. G., Bang, M. S., Kim, J. S., Chung, C. K., et al. (2015, August). A study on a robot arm driven by three-dimensional trajectories predicted from non-invasive neural signals. BioMedical Engineering OnLine,14(1), 81.Google Scholar
  31. 31.
    González-Mendoza, A., Pérez-Benítez, J. L., Pérez-Benítez, J. A., Espina-Hernández, J. H. (2015, February). Brain computer interface based on SSVEP for controlling a remote control car. In 2015 International Conference on Electronics, Communications and Computers (CONIELECOMP) (pp. 93–97).Google Scholar
  32. 32.
    Wu, C. M., Chen, Y. J., Zaeni, I. A. E., & Chen, S. C. (2016, November). A new SSVEP based BCI application on the mobile robot in a maze game. In 2016 International Conference on Advanced Materials for Science and Engineering (ICAMSE) (pp. 550–553).Google Scholar
  33. 33.
    Sharbrough, F., Chatrian, G. E., Lesser, R. P., Luders, H., Nuwer, M., & Picton, T. W. (1991). American Electroencephalographic Society guidelines for standard electrode position nomenclature. Journal of Clinical Neurophysiology, 8, 200–202.CrossRefGoogle Scholar
  34. 34.
    Schalk, G., McFarland, D. J., Hinterberger, T., Birbaumer, N., & Wolpaw, J. R. (2004, June). Bci2000: A general-purpose brain-computer interface (BCI) system. IEEE Transactions on Biomedical Engineering,51(6), 1034–1043.CrossRefGoogle Scholar
  35. 35.
    Wang, Y. Q. (2014). An analysis of the Viola-Jones face detection algorithm. Image Processing On Line, 4, 128–148.CrossRefGoogle Scholar
  36. 36.
    Neumann, D., Langner, T., Ulbrich, F., Spitta, D., & Goehring, D. (2017, June). Online vehicle detection using Haar-like, LBP and HOG feature based image classifiers with stereo vision preselection. In 2017 IEEE Intelligent Vehicles Symposium (IV) (pp. 773–778).Google Scholar
  37. 37.
    Kim, B., & Yu, S. C. (2017, February). Imaging sonar based real-time underwater object detection utilizing adaboost method. In 2017 IEEE Underwater Technology (UT) (pp. 1–5).Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  • G. Foresi
    • 1
  • A. Freddi
    • 1
  • S. Iarlori
    • 1
    Email author
  • S. Longhi
    • 1
  • A. Monteriù
    • 1
  • D. Ortenzi
    • 1
  • D. Proietti Pagnotta
    • 1
  1. 1.Università Politecnica delle MarcheAnconaItaly

Personalised recommendations