Evaluating ANN Efficiency in Recognizing EEG and Eye-Tracking Evoked Potentials in Visual-Game-Events

  • Andreas Wulff-JensenEmail author
  • Luis Emilio Bruni
Conference paper
Part of the Advances in Intelligent Systems and Computing book series (AISC, volume 586)


EEG and Eye-tracking signals have customarily been analyzed and inspected visually in order to be correlated to the controlled stimuli. This process has proven to yield valid results as long as the stimuli of the experiment are under complete control (e.g.: the order of presentation). In this study, we have recorded the subject’s electroencephalogram and eye-tracking data while they were exposed to a 2D platform game. In the game we had control over the design of each level by choosing the diversity of actions (i.e. events) afforded to the player. However we had no control over the order in which these actions were undertaken. The psychophysiological signals were synchronized to these game events and used to train and test an artificial neural network in order to evaluate how efficiently such a tool can help us in establishing the correlation, and therefore differentiating among the different categories of events. The highest average accuracies were between 60.25%–72.07%, hinting that it is feasible to recognize reactions to complex uncontrolled stimuli, like game events, using artificial neural networks.


Artificial neural network Machine learning Electroencephalogram Eye-tracking Games Pupillometry Game events Psychophysiology 


  1. 1.
    Koelstra, S., Mühl, C., Soleymani, M., Lee, J.S., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., Patras, I.: DEAP: a database for emotion analysis; using physiological signals. IEEE Trans. Affect. Comput. 3, 18–31 (2012). doi: 10.1109/T-AFFC.2011.15 CrossRefGoogle Scholar
  2. 2.
    Jain, A.K., Mao, J.: Artificial neural network: a tutorial. Communications 29, 31–44 (1996). doi: 10.1109/2.485891 Google Scholar
  3. 3.
    Basheer, I.A., Hajmeer, M.: Artificial neural networks: fundamentals, computing, design, and application. J. Microbiol. Methods 43, 3–31 (2000). doi: 10.1016/S0167-7012(00)00201-3 CrossRefGoogle Scholar
  4. 4.
    Andrews, R., Diederich, J., Tickle, A.B.: Survey and critique of techniques for extracting rules from trained artificial neural networks. Knowl.-Based Syst. 8, 373–389 (1995). doi: 10.1016/0950-7051(96)81920-4 CrossRefzbMATHGoogle Scholar
  5. 5.
    Kubacki, A., Jakubowski, A., Sawicki, Ł.: Detection of artefacts from the motion of the eyelids created during eeg research using artificial neural network. Robot. Meas. Tech. Adv. Intell. Syst. Comput. 440, 447–457 (2016). doi: 10.1007/978-3-319-29357-8_24 CrossRefGoogle Scholar
  6. 6.
    Wu, F.Y., Slater, J.D., Honig, L.S., Ramsay, R.E.: A neural network design for event-related potential diagnosis. Comput. Biol. Med. 23(3), 251–264 (1993)CrossRefGoogle Scholar
  7. 7.
    Turnip, A., Hong, K.S.: Classifying mental activities from EEG-P300 signals using adaptive neural networks. Int. J. Innov. Comput. Inf. Control 8, 6429–6443 (2012)Google Scholar
  8. 8.
    Gupta, L., Molfese, D.L., Ravi, T.: An artificial neural-network approach to ERP classification. Brain Cogn. 27, 311–330 (1995)CrossRefGoogle Scholar
  9. 9.
    Wilson, G.F., Christopher, R.A.: Real-time assessment of mental workload using psychophysiological measures and artificial neural networks. Hum. Factors 45, 635–643 (2004). doi: 10.1518/hfes.45.4.635.27088 CrossRefGoogle Scholar
  10. 10.
    Wilson, G., Russell, C.: Operator functional state classification using multiple psychophysiological features in an air traffic control task. Hum. Factors 45, 381–389 (2003)CrossRefGoogle Scholar
  11. 11.
    Baldwin, C.L., Penaranda, B.N.: Adaptive training using an artificial neural network and EEG metrics for within- and cross-task workload classification. NeuroImage 59, 48–56 (2012). doi: 10.1016/j.neuroimage.2011.07.047. Elsevier Inc.CrossRefGoogle Scholar
  12. 12.
    Robert, C., Gaudy, J.-F., Limoge, A.: Electroencephalogram processing using neural networks. Clin. Neurophysiol.: Off. J. Int. Fed. Clin. Neurophysiol. 113, 694–701 (2002). doi: 10.1016/S1388-2457(02)00033-0 CrossRefGoogle Scholar
  13. 13.
    Yannakakis, G.N., Hallam, J., Lund, H.H.: Entertainment capture through heart rate activity in physical interactive playgrounds. User Model. User-Adap. Interact. 18, 207–243 (2008). doi: 10.1007/s11257-007-9036-7 CrossRefGoogle Scholar
  14. 14.
    Perez Martínez, H., Garbarino, M., Yannakakis, G.N.: Generic physiological features as predictors of player experience. Affect. Comput. Intel. Interact. 267–276 (2011). doi: 10.1007/978-3-642-24600-5_30
  15. 15.
    Weber, R., Behr, K.M., Tamborini, R., Ritterfeld, U., Mathiak, K.: What do we really know about first-person-shooter games? An event-related, high-resolution content analysis. J. Comput.-Med. Commun. 14, 1016–1037 (2009). doi: 10.1111/j.1083-6101.2009.01479.x CrossRefGoogle Scholar
  16. 16.
    Togelius, J., Karakovskiy, S., Shaker, N.: MarioAI (2012). Accessed 14 Mar 2016
  17. 17.
    Shaker, N., Asteriadis, S., Yannakakis, G.N., Karpouzis, K.: A game-based corpus for analysing the interplay between game context and player experience. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), LNCS, vol. 6975, pp. 547–556 (2011). doi: 10.1007/978-3-642-24571-8_68
  18. 18.
    Shaker, N., Asteriadis, S., Yannakakis, G.N., Karpouzis, K.: Fusing visual and behavioral cues for modeling user experience in games. IEEE Trans. Cybern. 43, 1519–1531 (2013). doi: 10.1109/TCYB.2013.2271738 CrossRefGoogle Scholar
  19. 19.
    Shaker, N., Shaker, M., Abou-zleikha, M.: Towards generic models of player experience. In: Proceedings, the Eleventh AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (AIIDE-2015), 191–197 (2015)., ISBN 157735740X 9781577357407
  20. 20.
    Yannakakis, G.N., Togelius, J.: Experience-driven procedural content generation. IEEE Trans. Affect. Comput. 2(3), 147–161 (2011). doi: 10.1109/T-AFFC.2011.6 CrossRefGoogle Scholar
  21. 21.
    Togelius, J., Shaker, N., Karakovskiy, S., Yannakakis, G.N.: The Mario AI championship 2009-2012. AI Mag. 34, 89–92 (2013). doi: 10.1609/aimag.v34i3.2492 Google Scholar
  22. 22.
    The_EyeTribe: The Eye Tribe Tracker (2015)Google Scholar
  23. 23.
    G.tec, medical engineering: Advanced Biosignal Acquisition, Processing and Analysis Products 2013–2014, pp. 1–96 (2013)Google Scholar
  24. 24.
    Kowalik, M.: Do-it-yourself eye tracker: impact of the viewing angle on the eye tracking accuracy. In: Proceedings of CESCG 2011: The 15th Central European Seminar on Computer Graphics, pp. 1–7 (2011)., ISBN 978-3-9502533-3-7
  25. 25.
    Ooms, K.: Accuracy and precision of fixation locations recorded with the low-cost Eye Tribe tracker in different experimental set- ups 8 1–20 (2015). doi: 10.16910/jemr.8.1.5
  26. 26.
    Kosslyn, S.M.: Measuring the visual angle of the mind’s eye. Cogn. Psychol. 10, 356–389 (1978). doi: 10.1016/0010-0285(78)90004-X CrossRefGoogle Scholar
  27. 27.
    Key, A.P.F., Dove, G.O., Maguire, M.J.: Linking brainwaves to the brain: an ERP primer. Dev. Neuropsychol. 27, 183–215 (2005). doi: 10.1207/s15326942dn2702_1 CrossRefGoogle Scholar
  28. 28.
    Başar, E., Başar-Eroglu, C., Karakaş, S., Schürmann, M.: Are cognitive processes manifested in event-related gamma, alpha, theta and delta oscillations in the EEG? Neurosci. Lett. 259, 165–168 (1999). doi: 10.1016/S0304-3940(98)00934-3 CrossRefGoogle Scholar
  29. 29.
    Sprague, N., Ballard, D., Robinson, A.: Modeling embodied visual behaviors. ACM Trans. Appl. Percept. 4, 11 (2007). doi: 10.1145/1265957.1265960 CrossRefGoogle Scholar
  30. 30.
    Reilly, R.: Triangulating the reading brain: eye movements, computational models, and EEG. In: Current Trends in Eye Tracking Research, pp. 131–139. Springer International Publishing (2014)Google Scholar
  31. 31.
    Laeng, B., Sirois, S., Gredeback, G.: Pupillometry: a window to the preconscious? Perspect. Psychol. Sci. 7, 18–27 (2012). doi: 10.1177/1745691611427305 CrossRefGoogle Scholar
  32. 32.
    Hyvärinen, A., Oja, E.: Independent component analysis: algorithms and applications. Neural Netw. 13, 411–430 (2000). doi: 10.1016/S0893-6080(00)00026-5 CrossRefGoogle Scholar
  33. 33.
    Makeig, S., Bell, A.J., Jung, T.-P., Sejnowski, T.J.: Independent component analysis of electroencephalographic data. Adv. Neural. Inf. Process. Syst. 8, 145–151 (1996). doi: 10.1109/ICOSP.2002.1180091 Google Scholar
  34. 34.
    Shaker, N., Yannakakis, G.N., Togelius, J.: Towards automatic personalized content generation for platform games. In: AIIDE, October 2010.
  35. 35.
    Nogueira, P.A., Aguiar, R., Rodrigues, R., Oliveira, E.: Designing players’ emotional reaction models: a generic method towards adaptive affective gaming. In: 2014 9th Iberian Conference on Information Systems and Technologies (CISTI), pp. 1–6. IEEE. (2014). doi: 10.1109/CISTI.2014.6877079
  36. 36.
    D’Mello, S., Chipman, P., Graesser, A.: Posture as a predictor of learner’s affective engagement. In: Proceedings of the 29th Annual Meeting of the Cognitive Science Society, vol. 1 pp. 905–910 (2007)Google Scholar
  37. 37.
    MacDonald, J.H.: Kruskal-Wallis test. In: Biological Handbook of Statistics (2009)Google Scholar
  38. 38.
    Lindley, C.A., Sennersten., C.C.: Game play schemas: from player analysis to adaptive game mechanics. Int. J. Comput. Games Technol. 2008, 1–7 (2008). doi: 10.1155/2008/216784 CrossRefGoogle Scholar
  39. 39.
    Cecotti, H.: A time-frequency convolutional neural network for the offline classification of steady-state visual evoked potential responses. Pattern Recognit. Lett. 32 1145–1153 (2011). doi: 10.1016/j.patrec.2011.02.022
  40. 40.
    Heaton, J.: Encog: library of interchangeable machine learning models for Java and C#. J. Mach. Learn. Res. 16, 1243–1247 (2015)MathSciNetzbMATHGoogle Scholar
  41. 41.
    Kliegl, R., Dambacher, M., Dimigen, O., Sommer, W.: Oculomotor control, brain potentials, and timelines of word recognition during natural reading. In: Current Trends in Eye Tracking Research, pp. 141–155. Springer International Publishing (2014)Google Scholar

Copyright information

© Springer International Publishing AG 2018

Authors and Affiliations

  1. 1.Department of Architecture, Design and Media TechnologyAalborg University CopenhagenCopenhagenDenmark

Personalised recommendations