Skip to main content
SpringerLink
Log in
Book cover

Canadian Society of Civil Engineering Annual Conference

CSCE 2021: Proceedings of the Canadian Society of Civil Engineering Annual Conference 2021 pp 295–307Cite as

Enhanced Robotic Teleoperation in Construction Using a GAN-Based Physiological Signal Augmentation Framework

  • Conference paper
  • First Online:

Part of the book series: Lecture Notes in Civil Engineering ((LNCE,volume 239))

Abstract

Communication channels between humans and robots can alleviate safety hazards (collisions, interference) during human–robot collaboration (HRC) at construction sites. Recently, the authors demonstrated the feasibility of improving HRC by using electroencephalogram (EEG) signals to establish hands-free, nonverbal communication. Despite the potential of EEG to provide reliable means of communication, there is a concern regarding the quality of the collection of EEG signals, especially low spatial resolution. EEG signals collected from wearable devices suffer from this problem because the number of electrodes (5–32) is much lower than traditional clinical EEG systems (64–256). More importantly, the low spatial resolution may reduce the reliability of human–robot communication driven by EEG signals. To address this challenge, this study seeks to increase the spatial resolution of EEG signals by proposing a generative adversarial network (GAN)-based data augmentation framework. In it, artificial EEG signals will be produced from actual signals. To examine the feasibility of increasing the spatial resolution of EEG signals for an improved HRC, the EEG dataset of four subjects was collected using a wearable 32-channel device. The results show that the framework can enhance the spatial resolution of the collected dataset by 39.1% by generating realistic artificial signals. It also increased the accuracy of the EEG-based robotic teleoperation by 4.9%. It is expected that increasing the spatial resolution of EEG signals will improve the reliability of EEG-based human–robot communication.

This is a preview of subscription content, log in via an institution.

Chapter
USD   29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD   219.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD   279.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info
Hardcover Book
USD   279.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Learn about institutional subscriptions

References

  1. Ang KK, Chin ZY, Wang C, Guan C, Zhang H (2012) Filter bank common spatial pattern algorithm on BCI competition IV datasets 2a and 2b. Front Neurosc 6(MAR):1–9. https://doi.org/10.3389/fnins.2012.00039

  2. Bock T, Linner T (2016) Single-task construction robots by category. Constr Robots. Cambridge University Press, Cambridge, 14–290.https://doi.org/10.1017/CBO9781139872041.002

  3. Dumoulin V, Visin F (2016) A guide to convolution arithmetic for deep learning. arXiv, 1–31

    Google Scholar 

  4. Gevins A, Le J, Martin NK, Brickett P, Desmond J, Reutter B (1994) High resolution EEG: 124-channel recording, spatial deblurring and MRI integration methods. Electroencephalogr Clin Neurophysiol 90(5):337–358. https://doi.org/10.1016/0013-4694(94)90050-7

    Article  Google Scholar 

  5. Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A, Bengio Y (2014) Generative adversarial networks. Commun ACM 63(11):139–144. https://doi.org/10.1145/3422622

    Article  Google Scholar 

  6. Guger C, Edlinger G, Harkam W, Niedermayer I, Pfurtscheller G (2003) How many people are able to operate an EEG-based brain-computer interface (BCI)? IEEE Trans Neural Syst Rehabil Eng 11(2):145–147. https://doi.org/10.1109/TNSRE.2003.814481

    Article  Google Scholar 

  7. Gwin JT, Gramann K, Makeig S, Ferris DP (2010) Removal of movement artifact from high-density EEG recorded during walking and running. J Neurophysiol 103(6):3526–3534. https://doi.org/10.1152/jn.00105.2010

    Article  Google Scholar 

  8. Hartmann KG, Schirrmeister RT, Ball T (2018) EEG-GAN: Generative adversarial networks for electroencephalograhic (EEG) brain signals. arXiv.

    Google Scholar 

  9. Jebelli H, Hwang S, Lee S (2018) EEG signal-processing framework to obtain high-quality brain waves from an off-the-shelf wearable EEG device. J Comput Civ Eng 32(1):1–12. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000719

    Article  Google Scholar 

  10. Jebelli H, Hwang S, Lee S (2018) EEG-based workers’ stress recognition at construction sites. Autom Constr 93:315–324. https://doi.org/10.1016/j.autcon.2018.05.027

    Article  Google Scholar 

  11. Jebelli H, Khalili MM, Lee S (2019) Mobile eeg-based workers’ stress recognition by applying deep neural network. Advances in informatics and computing in civil and construction engineering, Springer International Publishing, Cham, 173–180.https://doi.org/10.1007/978-3-030-00220-6_21

  12. Jiao Y, Deng Y, Luo Y, Lu BL (2020) Driver sleepiness detection from EEG and EOG signals using GAN and LSTM networks. Neurocomputing, Elsevier B.V., 408, 100–111. https://doi.org/10.1016/j.neucom.2019.05.108

  13. Karras T, Aila T, Laine S, Lehtinen J (2017) “Progressive growing of gans for improved quality, stability, and variation. arXiv, 1–26

    Google Scholar 

  14. Kwon M, Han S, Kim K, Jun SC (2019) Super-resolution for improving EEG spatial resolution using deep convolutional neural network—feasibility study. Sensors (Switzerland) 19(23). https://doi.org/10.3390/s19235317

  15. Liu Y, Habibnezhad M, Jebelli H (2021a) Brain-computer interface for hands-free teleoperation of construction robots. Autom Const, Elsevier B.V., 123(November 2020), 103523. https://doi.org/10.1016/j.autcon.2020.103523

  16. Liu Y, Habibnezhad M, Jebelli H (2021b) Brainwave-driven human-robot collaboration in construction. Autom Constr, Elsevier B.V., 124(January), 103556. https://doi.org/10.1016/j.autcon.2021.103556

  17. Liu Y, Habibnezhad M, Jebelli H, Asadi S, Lee S (2020) Ocular Artifacts Reduction in EEG signals acquired at construction sites by applying a dependent component analysis (DCA). Constr Res Congress 2020, American Society of Civil Engineers, Reston, VA, 1281–1289.https://doi.org/10.1061/9780784482865.135

  18. Luo Y, Lu B-L (2018) Eeg data augmentation for emotion recognition using a conditional wasserstein GAN. 2018 40th Annual international conference of the IEEE engineering in medicine and biology society (EMBC), IEEE, 2535–2538. https://doi.org/10.1109/EMBC.2018.8512865

  19. Mao X, Li M, Li W, Niu L, Xian B, Zeng M, Chen G (2017) Progress in EEG-based brain robot interaction systems. Comput Intell Neurosci, 1–25.https://doi.org/10.1155/2017/1742862

  20. Petters S, Belden R (2014) SAM, the robotic bricklayer. SMART/Dyn Masonry 1:10–14

    Google Scholar 

  21. Peyré G, Cuturi M (2019) Computational optimal transport. Foundations Trends Mach Learn 11(5–6):1–257. https://doi.org/10.1561/2200000073

    Article  MATH  Google Scholar 

  22. Rahman M, Wang Y (2014) Dynamic emotion-based human-robot collaborative assembly in manufacturing, 1–4

    Google Scholar 

  23. Robla-Gomez S, Becerra VM, Llata JR, Gonzalez-Sarabia E, Torre-Ferrero C, Perez-Oria J (2017) Working together: a review on safe human-robot collaboration in industrial environments. IEEE Access 5:26754–26773. https://doi.org/10.1109/ACCESS.2017.2773127

    Article  Google Scholar 

  24. Salazar-Gomez AF, Delpreto J, Gil S, Guenther FH, Rus D (2017) Correcting robot mistakes in real time using EEG signals. Proceedings—IEEE international conference on robotics and automation, IEEE, 6570–6577.https://doi.org/10.1109/ICRA.2017.7989777

  25. Song J, Davey C, Poulsen C, Luu P, Turovets S, Anderson E, Li K, Tucker D (2015) EEG source localization: sensor density and head surface coverage. J Neurosc Methods, Elsevier B.V., 256:9–21. https://doi.org/10.1016/j.jneumeth.2015.08.015

  26. Tucker DM (1993) Spatial sampling of head electrical fields: the geodesic sensor net. Electroencephalogr Clin Neurophysiol 87(3):154–163. https://doi.org/10.1016/0013-4694(93)90121-B

    Article  Google Scholar 

  27. Val-Calvo M, Álvarez-Sánchez JR, Ferrández-Vicente JM, Fernández E (2019) Optimization of real-time EEG artifact removal and emotion estimation for human-robot interaction applications. Front Comput Neurosci. https://doi.org/10.3389/fncom.2019.00080

  28. Vasic M, Billard A (2013) Safety issues in human-robot interactions. 2013 IEEE international conference on robotics and automation, IEEE, 197–204. https://doi.org/10.1109/ICRA.2013.6630576

  29. Villani V, Pini F, Leali F, Secchi C (2018) Survey on human–robot collaboration in industrial settings: safety, intuitive interfaces and applications. Mechatronics, Elsevier 55(March):248–266. https://doi.org/10.1016/j.mechatronics.2018.02.009

    Article  Google Scholar 

  30. Xu Y, Ding C, Shu X, Gui K, Bezsudnova Y, Sheng X, Zhang D (2019) Shared control of a robotic arm using non-invasive brain–computer interface and computer vision guidance. Robot Auton Sys, Elsevier B.V., 115, 121–129. https://doi.org/10.1016/j.robot.2019.02.014

  31. Zhao L, Zhang Z (2012) Controlling method of industrial robots based on the electroencephalogram. CSAE 2012—proceedings, 2012 IEEE international conference on computer science and automation engineering, IEEE 3(2):566–569. https://doi.org/10.1109/CSAE.2012.6273016

Download references

Author information

Authors and Affiliations

  1. Department of Architectural Engineering and Department of Electrical Engineering and Computer Science, Pennsylvania State University, 201 Engineering Unit B, University Park, State College, PA, 20116802, USA

    Y. Liu

  2. Department of Architectural Engineering, Pennsylvania State University, 224 Engineering Unit A, University Park, State College, PA, 22416802, USA

    H. Jebelli

Authors
  1. Y. Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. H. Jebelli
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to H. Jebelli .

Editor information

Editors and Affiliations

  1. University of Waterloo, Waterloo, ON, Canada

    Scott Walbridge

  2. Concordia University, Montreal, QC, Canada

    Mazdak Nik-Bakht

  3. University of Regina, Regina, SK, Canada

    Kelvin Tsun Wai Ng

  4. Matrix Solutions Inc., Edmonton, AB, Canada

    Manas Shome

  5. University of British Columbia - Okanagan Campus, Kelowna, BC, Canada

    M. Shahria Alam

  6. University of Western Ontario, London, ON, Canada

    Ashraf el Damatty

  7. University of British Columbia - Okanagan Campus, Kelowna, BC, Canada

    Gordon Lovegrove

Rights and permissions

Reprints and permissions

Copyright information

© 2023 Canadian Society for Civil Engineering

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Liu, Y., Jebelli, H. (2023). Enhanced Robotic Teleoperation in Construction Using a GAN-Based Physiological Signal Augmentation Framework. In: Walbridge, S., et al. Proceedings of the Canadian Society of Civil Engineering Annual Conference 2021 . CSCE 2021. Lecture Notes in Civil Engineering, vol 239. Springer, Singapore. https://doi.org/10.1007/978-981-19-0503-2_24

Download citation

  • DOI: https://doi.org/10.1007/978-981-19-0503-2_24

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-19-0502-5

  • Online ISBN: 978-981-19-0503-2

  • eBook Packages: EngineeringEngineering (R0)

Publish with us

Policies and ethics

Search

Navigation