Advertisement

Swarm Intelligence

, Volume 10, Issue 4, pp 247–265 | Cite as

Electroencephalography as implicit communication channel for proximal interaction between humans and robot swarms

  • Luca Mondada
  • Mohammad Ehsanul Karim
  • Francesco Mondada
Article

Abstract

Search and rescue, autonomous construction, and many other semi-autonomous multirobot applications can benefit from proximal interactions between an operator and a swarm of robots. Most research on proximal interaction is based on explicit communication techniques such as gesture and speech. This study proposes a new implicit proximal communication technique to approach the problem of robot selection. We use electroencephalography (EEG) signals to select the robot at which the operator is looking. This is achieved using steady-state visually evoked potential (SSVEP), a repeatable neural response to a regularly blinking visual stimulus that varies predictively based on the blinking frequency. In our experiments, each robot was equipped with LEDs blinking at a different frequency, and the operator’s SSVEP neural response was extracted from the EEG signal to detect and select the robot without requiring any conscious action by the user. This study systematically investigates several parameters affecting the SSVEP neural response: blinking frequency of the LED, distance between the robot and the operator, and color of the LED. Based on these parameters, we study two signal processing approaches and critically analyze their performance on 10 subjects controlling a set of physical robots. Our results show that despite numerous artifacts, it is possible to achieve a recognition rate higher than 85 % on some subjects, while the average over the ten subjects was 75 %.

Keywords

Human–robots interaction EEG SSVEP Emotiv EPOC Thymio robot 

Notes

Acknowledgments

Many thanks to Dr. Ricardo Chavarriaga, Dr. Claire Braboszcz, and Dr. Serafeim Perdikis for the constructive discussions about experiments involving EEG; to Dr. Jérôme Scherer and Prof. Marco Picasso for their help on mathematical issues in the signal processing; to the reviewers who contributed with detailed and constructive comments during the submission process; and to all subjects who were available for the experiments. This work was partially supported by the Swiss National Center of Competence in Research “Robotics.”

References

  1. Akhtar, A., Norton, J. J., Kasraie, M., & Bretl, T. (2014). Playing checkers with your mind: An interactive multiplayer hardware game platform for brain–computer interfaces. In 36th Annual international conference of the IEEE engineering in medicine and biology society (EMBC), IEEE (pp. 1650–1653).Google Scholar
  2. Aljshamee, M., Mohammed, M. Q., Malekpour, A., Luksch, P., et al. (2014). Beyond pure frequency and phases exploiting: Color influence in SSVEP based on BCI. Computer Technology and Application, 5(2), 111–118.Google Scholar
  3. Aljshamee, M., Nadir, S., Malekpour, A., & Luksch, P. (2016). Discriminate the brain responses of multiple colors based on regular/irregular SSVEP paradigms. Journal of Medical and Bioengineering, 5(2), 89–92. doi: 10.18178/jomb.5.2.89-92.Google Scholar
  4. Beverina, F., Palmas, G., & Silvoni, S. (2003). User adaptive BCIs: SSVEP and P300 based interfaces. PsychNology, 1(4), 331–354.Google Scholar
  5. Bi, L., Fan, X. A., & Liu, Y. (2013). EEG-based brain-controlled mobile robots: A survey. IEEE Transactions on Human-Machine Systems, 43(2), 161–176. doi: 10.1109/TSMCC.2012.2219046.CrossRefGoogle Scholar
  6. Bonani, M., Rétornaz, P., Magnenat, S., Bleuler, H., & Mondada, F. (2012). Physical interactions in swarm robotics: The hand-bot case study. In A. Martinoli, F. Mondada, N. Correll, G. Mermoud, M. Egerstedt, M. A. Hsieh, L. E. Parker, & K. Støy (Eds.), Distributed autonomous robotic systems, Springer tracts in advanced robotics (pp. 585–595). Berlin: Springer.Google Scholar
  7. Brambilla, M., Ferrante, E., Birattari, M., & Dorigo, M. (2013). Swarm robotics: A review from the swarm engineering perspective. Swarm Intelligence, 7(1), 1–41. doi: 10.1007/s11721-012-0075-2.CrossRefGoogle Scholar
  8. Cao, T., Wan, F., Mak, P. U., Mak, P. I., Vai, M. I., & Hu, Y. (2012). Flashing color on the performance of SSVEP-based brain–computer interfaces. In Annual international conference of the IEEE engineering in medicine and biology society (EMBC), IEEE (pp. 1819–1822).Google Scholar
  9. Chua, F., Daftari, A., Alvarez, T., DeMarco, R., Bergen, M., Beck, K., & Servatius, R. (2004). Effects of a single green flash versus a white flash of light on saccadic oculomotor metrics. In Proceedings of the IEEE 30th annual northeast bioengineering conference, IEEE (pp. 9–10).Google Scholar
  10. Couture-Beil, A., Vaughan, R., & Mori, G. (2010). Selecting and commanding individual robots in a vision-based multi-robot system. In 5th ACM/IEEE international conference on human–robot interaction (HRI) (pp. 355–356). doi: 10.1109/HRI.2010.5453167.
  11. Durak, L., & Arikan, O. (2003). Short-time fourier transform: Two fundamental properties and an optimal implementation. IEEE Transactions on Signal Processing, 51(5), 1231–1242. doi: 10.1109/TSP.2003.810293.MathSciNetCrossRefGoogle Scholar
  12. Duvinage, M., Castermans, T., Petieau, M., Hoellinger, T., Cheron, G., & Dutoit, T. (2013). Performance of the emotiv EPOC headset for P300-based applications. Biomedical Engineering Online, 12(1), 56.CrossRefGoogle Scholar
  13. Faller, J., Müller-Putz, G., Schmalstieg, D., & Pfurtscheller, G. (2010). An application framework for controlling an avatar in a desktop-based virtual environment via a software SSVEP brain computer interface. Presence: Teleoperators and Virtual Environments, 19(1), 25–34. doi: 10.1162/pres.19.1.25.CrossRefGoogle Scholar
  14. Fan, X., Bi, L., Teng, T., Ding, H., & Liu, Y. (2015). A brain–computer interface-based vehicle destination selection system using P300 and SSVEP signals. IEEE Transactions on Intelligent Transportation Systems, 16(1), 274–283.CrossRefGoogle Scholar
  15. Fong, T., Thorpe, C., & Baur, C. (2003). Multi-robot remote driving with collaborative control. IEEE Transactions on Industrial Electronics, 50(4), 699–704. doi: 10.1109/TIE.2003.814768.CrossRefGoogle Scholar
  16. Gao, X., Xu, D., Cheng, M., & Gao, S. (2003). A BCI-based environmental controller for the motion-disabled. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 11(2), 137–140. doi: 10.1109/TNSRE.2003.814449.CrossRefGoogle Scholar
  17. Goodrich, M. A., Pendleton, B., Kerman, S., & Sujit, P. (2013). What types of interactions do bio-inspired robot swarms and flocks afford a human? In Proceedings of robotics science and systems VIII (pp. 105–112).Google Scholar
  18. Goodrich, M. A., & Schultz, A. C. (2007). Human–robot interaction: A survey. Foundations and Trends in Human Computer Interaction, 1(3), 203–275. doi: 10.1561/1100000005.CrossRefzbMATHGoogle Scholar
  19. Guger, C., Allison, B., Grosswindhager, B., Prückl, R., Hintermüller, C., Kapeller, C., et al. (2012). How many people could use an SSVEP BCI? Frontiers in Neuroscience, 6, 169. doi: 10.3389/fnins.2012.00169.Google Scholar
  20. Güneysu, A., & Akin, H. L. (2013). An SSVEP based BCI to control a humanoid robot by using portable EEG device. In International conference of the IEEE engineering in medicine and biology society, (Vol. 2013, pp. 6905–6908). doi: 10.1109/EMBC.2013.6611145.
  21. Herrmann, C. S. (2001). Human EEG responses to 1–100 Hz flicker: Resonance phenomena in visual cortex and their potential correlation to cognitive phenomena. Experimental Brain Research, 137, 346–353. doi: 10.1007/s002210100682.CrossRefGoogle Scholar
  22. Hvaring, F. T., & Ulltveit-Moe, A. H. (2014). A comparison of visual evoked potential (VEP)-based methods for the low-cost emotiv EPOC neuroheadset. Technical report, Norwegian University of Science and Technology, NTNU—Trondheim, Trondheim, Norway. http://www.diva-portal.org/smash/get/diva2:751718/FULLTEXT01.pdf.
  23. Jacobs, G. (2013). SSVEP-based BCI control for navigating a robot. Technical Report. December, Faculteit der Sociale Wetenschappen, Radboud University, Nijmegen, The Netherlands. http://www.ru.nl/publish/pages/641151/jacobs_gj_ba-th-2013.pdf.
  24. Jian, H. L., & Tang, K. T. (2014). Improving classification accuracy of SSVEP based BCI using RBF SVM with signal quality evaluation. In 2014 International symposium on intelligent signal processing and communication systems (ISPACS), IEEE (pp. 302–306).Google Scholar
  25. Jones, G., Berthouze, N., Bielski, R., & Julier, S. (2010). Towards a situated, multimodal interface for multiple UAV control. In 2010 IEEE international conference on robotics and automation (ICRA) (pp. 1739–1744). doi: 10.1109/ROBOT.2010.5509960.
  26. Kernbach, S. (2013). Handbook of collective robotics: Fundamentals and challenges. Boca Raton: CRC Press. doi: 10.4032/9789814364119.CrossRefGoogle Scholar
  27. Kirchner, E. A., de Gea Fernandez, J., Kampmann, P., Schröer, M., Metzen, J. H., & Kirchner, F. (2015). Intuitive interaction with robots—technical approaches and challenges. In R. Drechsler & U. Khne (Eds.), Formal modeling and verification of cyber-physical systems: 1st International summer school on methods and tools for the design of digital systems, Bremen, Germany, September 2015, Springer Fachmedien Wiesbaden, Wiesbaden (pp. 224–248). doi: 10.1007/978-3-658-09994-78.
  28. Kishore, S., González-Franco, M., Hintemüller, C., Kapeller, C., Guger, C., Slater, M., et al. (2014). Comparison of SSVEP BCI and eye tracking for controlling a humanoid robot in a social environment. Presence: Teleoperators and Virtual Environments, 23(3), 242–252.CrossRefGoogle Scholar
  29. Kolling, A., Walker, P., Chakraborty, N., Sycara, K., & Lewis, M. (2016). Human interaction with robot swarms: A survey. IEEE Transactions on Human-Machine Systems, 46(1), 9–26. doi: 10.1109/THMS.2015.2480801.CrossRefGoogle Scholar
  30. Li, Y., Pan, J., Wang, F., & Yu, Z. (2013). A hybrid bci system combining P300 and SSVEP and its application to wheelchair control. IEEE Transactions on Biomedical Engineering, 60(11), 3156–3166.CrossRefGoogle Scholar
  31. Lin, Y. P., Wang, Y., & Jung, T. P. (2014). Assessing the feasibility of online SSVEP decoding in human walking using a consumer EEG headset. Journal of NeuroEngineering and Rehabilitation, 11(1), 119. doi: 10.1186/1743-0003-11-119.CrossRefGoogle Scholar
  32. Liu, Y., Jiang, X., Cao, T., Wan, F., Mak, P. U., Mak, P. I., & Vai, M. I. (2012). Implementation of SSVEP based BCI with emotiv EPOC. In IEEE international conference on virtual environments human–computer interfaces and measurement systems (VECIMS), IEEE (pp. 34–37).Google Scholar
  33. Martinoli, A., Mondada, F., Correll N., Mermoud, G., Egerstedt, M., Hsieh, M. A, Parker, L. E., & Støy, K. (2012). 10th International symposium on distributed autonomous robotic systems (DARS) (Vol. 83). Springer.Google Scholar
  34. Mathews, N., Valentini, G., Christensen, A. L., O’Grady, R., Brutschy, A., & Dorigo, M. (2015). Spatially targeted communication in decentralized multirobot systems. Autonomous Robots, 38(4), 439–457.CrossRefGoogle Scholar
  35. Millan, J. R., Renkens, F., Mouriño, J., & Gerstner, W. (2004). Noninvasive brain-actuated control of a mobile robot by human EEG. IEEE Transactions on biomedical Engineering, 51(6), 1026–1033.CrossRefGoogle Scholar
  36. Monajjemi, V., Wawerla, J., Vaughan, R., & Mori, G. (2013). HRI in the sky: Creating and commanding teams of UAVs with a vision–mediated gestural interface. In IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 617–623). doi: 10.1109/IROS.2013.6696415.
  37. Mouli, S., Palaniappan, R., Sillitoe, I. P., & Gan, J. Q. (2013). Performance analysis of multi-frequency SSVEP-BCI using clear and frosted colour led stimuli. In Bioinformatics and bioengineering (BIBE), 2013 IEEE 13th international conference on, IEEE (pp. 1–4).Google Scholar
  38. Nagi, J., Giusti, A., Gambardella, L. M., & Di Caro, G. (2014). Human–swarm interaction using spatial gestures. In IEEE/RSJ international conference on intelligent robots and systems (IROS 2014) (pp. 3834–3841). doi: 10.1109/IROS.2014.6943101.
  39. Ortner, R., Guger, C., Prueckl, R., Grünbacher, E., & Edlinger, G. (2010). SSVEP based brain–computer interface for robot control. In International conference on computers for handicapped persons, Springer, Vol. 6180 LNCS (pp. 85–90).Google Scholar
  40. Pourmehr, S., Monajjemi, V., Vaughan, R., & Mori, G. (2013). “You two! Take off!”: Creating, modifying and commanding groups of robots using face engagement and indirect speech in voice commands. In IEEE/RSJ international conference on intelligent robots and systems (IROS) (pp. 137–142). doi: 10.1109/IROS.2013.6696344.
  41. Renard, Y., Lotte, F., Gibert, G., Congedo, M., Maby, E., Delannoy, V., et al. (2010). OpenViBE: An open-source software platform to design, test, and use brain–computer interfaces in real and virtual environments. Presence: Teleoperators and Virtual Environments, 19(1), 35–53. doi: 10.1162/pres.19.1.35.CrossRefGoogle Scholar
  42. Rencher, A. C. (2003). Methods of multivariate analysis (Vol. 492). New York: Wiley. doi: 10.1002/0471271357.zbMATHGoogle Scholar
  43. Riedo, F., Chevalier, M., Magnenat, S., Mondada, F. (2013). Thymio II, a robot that grows wiser with children. In IEEE workshop on advanced robotics and its social impacts (ARSO), IEEE (pp. 187–193). doi: 10.1109/ARSO.2013.6705527.
  44. Rubenstein, M., Ahler, C., & Nagpal, R. (2012). Kilobot: A low cost scalable robot system for collective behaviors. In IEEE international conference on robotics and automation (ICRA), IEEE (pp. 3293–3298).Google Scholar
  45. Rubenstein, M., Cornejo, A., & Nagpal, R. (2014). Programmable self-assembly in a thousand-robot swarm. Science, 345(6198), 795–799.CrossRefGoogle Scholar
  46. Rzepecki, J., Delcourt, J., Da Silva, M. P., & Le Callet, P. (2012). Virtual interactions: Can EEG help make the difference with real interaction? In Proceedings of the 2012 IEEE international conference on multimedia and expo workshops, ICMEW 2012 (pp. 151–156). doi: 10.1109/ICMEW.2012.33.
  47. Salehuddin, M., Suprijanto, S., & Muchtadi, F. I. (2011). Prototype design of low cost four channels digital electroencephalograph for sleep monitoring. In 2nd International conference on instrumentation control and automation (ICA) (pp. 188–193). doi: 10.1109/ICA.2011.6130154.
  48. Stawicki, P., Gembler, F., & Volosyak, I. (2016). Driving a semiautonomous mobile robotic car controlled by an SSVEP-based BCI. Computational Intelligence and Neuroscience. doi: 10.1155/2016/4909685.
  49. Stytsenko, K., Jablonskis, E., & Prahm, C. (2011). Evaluation of consumer EEG device emotiv EPOC. In MEi: CogSci conference 2011, Ljubljana.Google Scholar
  50. Trovato, G., Zecca, M., Sessa, S., Jamone, L., Ham, J., Hashimoto, K., et al. (2013). Cross-cultural study on human–robot greeting interaction: Acceptance and discomfort by egyptians and japanese. Paladyn, Journal of Behavioral Robotics, 4(2), 83–93. doi: 10.2478/pjbr-2013-0006.CrossRefGoogle Scholar
  51. Van Vliet, M., Robben, A., Chumerin, N., Manyakov, N. V., Combaz, A., & Van Hulle, M. M. (2012). Designing a brain–computer interface controlled video-game using consumer grade EEG hardware. In Biosignals and biorobotics conference (BRC), 2012, IEEE (pp. 1–6).Google Scholar
  52. Volosyak, I., Cecotti, H., & Graser, A. (2009). Optimal visual stimuli on LCD screens for SSVEP based brain–computer interfaces. In Neural Engineering (pp. 447–450).Google Scholar
  53. Winfield, A. F., & Nembrini, J. (2006). Safety in numbers: Fault-tolerance in robot swarms. International Journal of Modelling, Identification and Control, 1(1), 30–37.CrossRefGoogle Scholar
  54. Wu, C. H., & Lakany, H. (2013). The effect of the viewing distance of stimulus on SSVEP response for use in brain–computer interfaces. In IEEE international conference on systems, man, and cybernetics (SMC), IEEE (pp. 1840–1845).Google Scholar
  55. Yanco, H. A., & Drury, J. L. (2004). Classifying human–robot interaction: An updated taxonomy. In IEEE international conference on systems, man and cybernetics, SMC (pp. 2841–2846). doi: 10.1109/ICSMC.2004.1400763.
  56. Yin, E., Zeyl, T., Saab, R., Chau, T., Hu, D., & Zhou, Z. (2015). A hybrid brain–computer interface based on the fusion of P300 and SSVEP scores. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 23(4), 693–701.CrossRefGoogle Scholar
  57. Yin, E., Zhou, Z., Jiang, J., Chen, F., Liu, Y., & Hu, D. (2013). A novel hybrid BCI speller based on the incorporation of SSVEP into the P300 paradigm. Journal of Neural Engineering, 10(2), 026,012.CrossRefGoogle Scholar
  58. Zheng, Y., & Zhang, Y. (2010). An improved segmented match filters with FFT approach for GNSS signal acquisition. In ICCTD 2010—Proceedings of the 2nd international conference on computer technology and development, ICCTD (pp. 425–428). doi: 10.1109/ICCTD.2010.5645833.
  59. Zhu, D., Bieger, J., Molina, G. G., & Aarts, R. M. (2010). A survey of stimulation methods used in SSVEP-based BCIs. Computational Intelligence and Neuroscience, 702, 357. doi: 10.1155/2010/702357.Google Scholar

Copyright information

© Springer Science+Business Media New York 2016

Authors and Affiliations

  1. 1.Department of PhysicsSwiss Federal Institute of Technology ETHZZürichSwitzerland
  2. 2.Laboratoire de Systèmes RobotiquesEcole Polytechnique Fédérale de LausanneLausanneSwitzerland

Personalised recommendations