Multimedia Tools and Applications

, Volume 74, Issue 22, pp 10137–10159 | Cite as

Improving object segmentation by using EEG signals and rapid serial visual presentation

  • Eva Mohedano
  • Graham Healy
  • Kevin McGuinness
  • Xavier Giró-i-Nieto
  • Noel E. O’Connor
  • Alan F. Smeaton


This paper extends our previous work on the potential of EEG-based brain computer interfaces to segment salient objects in images. The proposed system analyzes the Event Related Potentials (ERP) generated by the rapid serial visual presentation of windows on the image. The detection of the P300 signal allows estimating a saliency map of the image, which is used to seed a semi-supervised object segmentation algorithm. Thanks to the new contributions presented in this work, the average Jaccard index was improved from 0.47 to 0.66 when processed in our publicly available dataset of images, object masks and captured EEG signals. This work also studies alternative architectures to the original one, the impact of object occupation in each image window, and a more robust evaluation based on statistical analysis and a weighted F-score.


Brain-computer interfaces Electroencephalography Rapid serial visual presentation Object segmentation Interactive segmentation GrabCut algorithm 



This publication has emanated from research conducted with the financial support of Science Foundation Ireland (SFI) under grant number SFI/12/RC/2289 and partially funded by the Project TEC2013-43935-R BigGraph of the Spanish Government.


  1. 1.
    Bauer G, Gerstenbrand F, Rumpl E (1979) Varieties of the locked-in syndrome. J Neurol 221(2):77–91CrossRefGoogle Scholar
  2. 2.
    Bell CJ, Shenoy P, Chalodhorn R, Rao R (2008) Control of a humanoid robot by a noninvasive brain computer interface in humans. J Neural Eng 16(5):432–441Google Scholar
  3. 3.
    Bigdely-Shamlo N, Vankov A, Ramirez R, Makeig S (2008) Brain activity-based image classification from rapid serial visual presentation. IEEE Trans Neural Syst Rehabil Eng 16(5):432–441CrossRefGoogle Scholar
  4. 4.
    Bradski G (2000) Dr. Dobb’s Journal of Software ToolsGoogle Scholar
  5. 5.
    Cruse D, Chennu S, Chatelle C, Bekinschtein TA, Fernández-Espejo D, Pickard JD, Laureys S, Owen AM (2012) Bedside detection of awareness in the vegetative state: a cohort study. Lancet 378(9809):2088–2094CrossRefGoogle Scholar
  6. 6.
    Everingham M, Van Gool L, Williams CKI, Winn J, Zisserman A (2010) The Pascal visual object classes (VOC) challenge. Int J Comput Vis 88(2):303–338CrossRefGoogle Scholar
  7. 7.
    Fernandez-Canellas D (2013) Modeling the temporal dependency of brain responses to rapidly presented stimuli in erp based bci. Master’s thesis, Northeastern UniversityGoogle Scholar
  8. 8.
    Healy G, Smeaton A (2011) Eye fixation related potentials in a target search task. In: Engineering in Medicine and Biology Society, EMBC, 2011 Annual International Conference of the IEEE, pp 4203–4206Google Scholar
  9. 9.
    Healy G, Smeaton AF (2011) Optimising the number of channels in eeg-augmented image search. In: Proceedings of the 25th BCS conference on human-computer interaction, BCS-HCI, pp 157–162Google Scholar
  10. 10.
    Hebbalaguppe R, McGuinness K, Kuklyte J, Healy G, Connor NO, Smeaton A (2013) How Interaction Methods Affect Image Segmentation : User Experience in the Task. In: Proc. The 1st IEEE workshop on user-centred computer vision (UCCV)Google Scholar
  11. 11.
    Hu X, Li K, Han J, Hua X, Guo L, Liu T (2012) Bridging the semantic gap via functional brain imaging. IEEE Trans Multimed 14(2):314–325CrossRefGoogle Scholar
  12. 12.
    Huang Y, Erdogmus D, Pavel M, Mathan S, Hild II KE (2011) A framework for rapid visual image search using single-trial brain evoked responses. Neurocomputing 74(12-13):2041–2051CrossRefGoogle Scholar
  13. 13.
    Kapoor A, Shenoy P, Tan D (2008) Combining brain computer interfaces with vision for object categorization. In: Computer vision and pattern recognition (CVPR), pp 1–8Google Scholar
  14. 14.
    Luck SJ (2005) An introduction to the event-related potential technique. MIT PressGoogle Scholar
  15. 15.
    Margolin R, Zelnik-Manor L, Tal A (2014) How to evaluate foreground maps?. In: CVPRGoogle Scholar
  16. 16.
    Martin D, Fowlkes C, Tal D, Malik J (2001) A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: ICCV, vol 2, pp 416–423Google Scholar
  17. 17.
    Mohedano E, Healy G, McGuinness K, Giró-i Nieto X, O’Connor NE, Smeaton AF (2014) Object segmentation in images using eeg signals. In: Proceedings of the ACM international conference on multimedia, MM’14. ACM, New York, NY, USA, pp 417–426Google Scholar
  18. 18.
    Motomura S, Ojima Y, Zhong N (2009) Eeg/erp meets act-r: A case study for investigating human computation mechanism. In: Zhong N, Li K, Lu S, Chen L (eds) Brain Informatics, volume 5819 of Lecture Notes in Computer Science, pp 63–73Google Scholar
  19. 19.
    Pathirage I, Khokar K, Klay E, Alqasemi R, Dubey R (2013) A vision based p300 brain computer interface for grasping using a wheelchair-mounted robotic arm. In: 2013 IEEE/ASME international conference on advanced intelligent mechatronics (AIM), pp 188–193Google Scholar
  20. 20.
    Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, Blondel M, Prettenhofer P, Weiss R, Dubourg V, Vanderplas J, Passos A, Cournapeau D, Brucher M, Perrot M, Duchesnay E (2011) Scikit-learn: Machine learning in Python. J Mach Learn Res 12:2825–2830zbMATHMathSciNetGoogle Scholar
  21. 21.
    Roark B, Oken B, M F-O, Orhan U, Erdogmus D (2013) Offline analysis of context contribution to erp-based typing bci performance. J Neural Eng 10(6):432–441Google Scholar
  22. 22.
    Rother C, Kolmogorov V, Blake A (2004) “GrabCut”: Interactive foreground extraction using iterated graph cuts. ACM Trans Graph 23(3):309–314CrossRefGoogle Scholar
  23. 23.
    Sajda P, Pohlmeyer E, Wang J, Parra LC, Christoforou C, Dmochowski J, Hanna B, Bahlmann C, Singh MK, Chang S-F (2010) In a blink of an eye and a switch of a transistor: cortically coupled computer vision. Proc IEEE 98(3):462–478CrossRefGoogle Scholar
  24. 24.
    Spence R (2002) Rapid, Serial and Visual: a presentation technique with potential. Inf Vis 1(1):13–19CrossRefGoogle Scholar
  25. 25.
    Wang J, Pohlmeyer E, Hanna B, Jiang Y-G, Sajda P, Chang S-F (2009) Brain state decoding for rapid image retrieval. In: Proceedings of the 17th ACM international conference on multimedia MM ’09, pp 945–954Google Scholar
  26. 26.
    Yazdani A, Vesin J-M, Izzo D, Ampatzis C, Ebrahimi T (2010) Implicit retrieval of salient images using brain computer interface. In: ICIP, pp 3169–3172Google Scholar

Copyright information

© Springer Science+Business Media New York 2015

Authors and Affiliations

  • Eva Mohedano
    • 1
  • Graham Healy
    • 1
  • Kevin McGuinness
    • 1
  • Xavier Giró-i-Nieto
    • 2
  • Noel E. O’Connor
    • 1
  • Alan F. Smeaton
    • 1
  1. 1.Insight Center for data AnalyticsDublin City UniversityDublinIreland
  2. 2.Image Processing GroupUniversitat Politcnica de CatalunyaCatalunyaSpain

Personalised recommendations