On the Use of Brain Decoded Signals for Online User Adaptive Gesture Recognition Systems

  • Kilian Förster
  • Andrea Biasiucci
  • Ricardo Chavarriaga
  • José del R. Millán
  • Daniel Roggen
  • Gerhard Tröster
Part of the Lecture Notes in Computer Science book series (LNCS, volume 6030)


Activity and context recognition in pervasive and wearable computing ought to continuously adapt to changes typical of open-ended scenarios, such as changing users, sensor characteristics, user expectations, or user motor patterns due to learning or aging. System performance inherently relates to the user’s perception of the system behavior. Thus, the user should be guiding the adaptation process. This should be automatic, transparent, and unconscious.

We capitalize on advances in electroencephalography (EEG) signal processing that allow for error related potentials (ErrP) recognition. ErrP are emitted when a human observes an unexpected behavior in a system. We propose and evaluate a hand gesture recognition system from wearable motion sensors that adapts online by taking advantage of ErrP. Thus the gesture recognition system becomes self-aware of its performance, and can self-improve through re-occurring detection of ErrP signals.

Results show that our adaptation technique can improve the accuracy of a user independent gesture recognition system by 13.9% when ErrP recognition is perfect. When ErrP recognition errors are factored in, recognition accuracy increases by 4.9%. We characterize the boundary conditions of ErrP recognition guaranteeing beneficial adaptation. The adaptive algorithms are applicable to other forms of activity recognition, and can also use explicit user feedback rather than ErrP.


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. 1.
    Ward, J.A.: Activity Monitoring: Continuous Recognition and Performance Evaluation. PhD thesis, ETH Zurich, Nr. 16520 (2006)Google Scholar
  2. 2.
    Davies, N., Siewiorek, D.P., Sukthankar, R.: Special issue: Activity-based computing. IEEE Pervasive Computing 7(2), 20–21 (2008)CrossRefGoogle Scholar
  3. 3.
    Ravi, N., Dandekar, N., Mysore, P., Littman, M.L.: Activity recognition from accelerometer data. American Association for Artificial Intelligence (2005)Google Scholar
  4. 4.
    Stiefmeier, T., Roggen, D., Ogris, G., Lukowicz, P., Tröster, G.: Wearable activity tracking in car manufacturing. IEEE Pervasive Computing Magazine 7(2), 42–50 (2008)CrossRefGoogle Scholar
  5. 5.
    Bao, L., Intille, S.S.: Activity recognition from user-annotated acceleration data. In: Pervasive Computing: Proc. of the 2nd Int Conference, pp. 1–17 (2004)Google Scholar
  6. 6.
    Lester, J., Choudhury, T., Borriello, G.: A practical approach to recognizing physical activities. In: Fishkin, K.P., Schiele, B., Nixon, P., Quigley, A. (eds.) PERVASIVE 2006. LNCS, vol. 3968, pp. 1–16. Springer, Heidelberg (2006)CrossRefGoogle Scholar
  7. 7.
    Nieuwenhuis, S., Ridderinkhof, K.R., Blom, J., Band, G.P., Kok, A.: Error-related brain potentials are differentially related to awareness of response errors: Evidence from an antisaccade task. Psychophysiology 38(5), 752–760 (2001)CrossRefGoogle Scholar
  8. 8.
    Yasuda, A., Sato, A., Miyawaki, K., Kumano, H., Kuboki, T.: Error-related negativity reflects detection of negative reward prediction error. Neuroreport 15(16), 2561–2565 (2004)CrossRefGoogle Scholar
  9. 9.
    Frank, M.J., Woroch, B.S., Curran, T.: Error-related negativity predicts reinforcement learning and conflict biases. Neuron 47(4), 495–501 (2005)CrossRefGoogle Scholar
  10. 10.
    Santosh, K.C., Nattee, C.: A comprehensive survey on on-line handwriting recognition technology and its real application to the nepalese natural handwriting (2009)Google Scholar
  11. 11.
    Tang, Y., Rose, R.: Rapid speaker adaptation using clustered maximum-likelihood linear basis with sparse training data. IEEE Transactions on Audio, Speech, and Language Processing 16(3), 607–616 (2008)CrossRefGoogle Scholar
  12. 12.
    Baker, J.M., Deng, L., Glass, J., Khudanpur, S., Lee, C.-H., Morgan, N., OShaughnessy, D.: Research developments and directions in speech recognition and understanding, part 1. IEEE Signal Processing Magazine 26(3), 75–80 (2009)CrossRefGoogle Scholar
  13. 13.
    Ohmura, R., Hashida, N., Imai, M.: Preliminary evaluation of personal adaptation techniques in accelerometer-based activity recognition. In: Proc. 13th IEEE Int. Symposium on Wearable Computers: Late Breaking Results (2009)Google Scholar
  14. 14.
    He, X., Zhao, Y.: Fast model selection based speaker adaptation for nonnative speech. IEEE Trans. on Speech and Audio Processing 11(4), 298–307 (2003)CrossRefMathSciNetGoogle Scholar
  15. 15.
    Kunze, K., Lukowicz, P.: Using acceleration signatures from everyday activities for on-body device location. In: 2007 11th IEEE International Symposium on Wearable Computers, September 2007, pp. 115–116 (2007)Google Scholar
  16. 16.
    Förster, K., Roggen, D., Tröster, G.: Unsupervised classifier self-calibration through repeated context occurences: Is there robustness against sensor displacement to gain? In: Proc. 13th IEEE Int. Symposium on Wearable Computers (ISWC), pp. 77–84 (2009)Google Scholar
  17. 17.
    Taylor, S.F., Stern, E.R., Gehring, W.J.: Neural systems for error monitoring: Recent findings and theoretical perspectives. Neuroscientist 13(2), 160–172 (2007)CrossRefGoogle Scholar
  18. 18.
    Falkenstein, M., Hoormann, J., Christ, S., Hohnsbein, J.: ERP components on reaction errors and their functional significance: A tutorial. Biol. Psychol. 51(2-3), 87–107 (2000)CrossRefGoogle Scholar
  19. 19.
    Ferrez, P.W., Millán, J.: Error-related EEG potentials generated during simulated brain-computer interaction. IEEE Trans. Biomed. Eng. 55, 923–929 (2008)CrossRefGoogle Scholar
  20. 20.
    Schalk, G., Wolpaw, J.R., McFarland, D.J., Pfurtscheller, G.: EEG-based communication: Presence of an error potential. Clin. Neurophysiol. 111(12), 2138–2144 (2000)CrossRefGoogle Scholar
  21. 21.
    Parra, L.C., Spence, C.D., Gerson, A.D., Sajda, P.: Response error correction–A demonstration of improved human-machine performance using real-time EEG monitoring. IEEE Trans. Neural. Syst. Rehabil. Eng. 11(2), 173–177 (2003)CrossRefGoogle Scholar
  22. 22.
    Fatourechi, M., Bashashati, A., Ward, R.K., Birch, G.E.: EMG and EOG artifacts in brain computer interface systems: A survey. Clin. Neurophysiol. 118(3), 480–494 (2007)CrossRefGoogle Scholar
  23. 23.
    Chavarriaga, R., Ferrez, P.W., Millán, J.: To Err Is Human: Learning from error potentials in brain-computer interfaces. In: International Conference on Cognitive Neurodynamics (2007)Google Scholar
  24. 24.
    Bollon, J.M., Chavarriaga, R., Millán, J., Bessière, P.: EEG error-related potentials detection with a Bayesian filter. In: 4th International IEEE EMBS Conference on Neural Engineering, Antalya Turkey (2009)Google Scholar
  25. 25.
    Gehring, W.J., Goss, B., Coles, M.G.H., Meyer, D.E., Donchin, E.A.: Neural system for error-detection and compensation. Psychol. Sci. 4, 385–390 (1993)CrossRefGoogle Scholar
  26. 26.
    Schlögl, A., Keinrath, C., Zimmermann, D., Scherer, R., Leeb, R., Pfurtscheller, G.: A fully automated correction method of EOG artifacts in EEG recordings. Clin. Neurophysiol. 118(1), 98–104 (2007)CrossRefGoogle Scholar
  27. 27.
    Liu, H., Setiono, R.: A probabilistic approach to feature selection - a filter solution, pp. 319–327. Morgan Kaufmann, San FranciscoGoogle Scholar
  28. 28.
    García Lopez, F., García Torres, M., Melian Batista, B., Moreno Perez, J.A., Moreno-Vega, J.M.: Solving feature subset selection problem by a parallel scatter search. European Journal of Operational Research 169(2), 477–489 (2006)MATHCrossRefMathSciNetGoogle Scholar
  29. 29.
    John, G., Langley, P.: Estimating continuous distributions in Bayesian classifiers. In: Proceedings of the Eleventh Conference on Uncertainty in Artificial Intelligence, pp. 338–345. Morgan Kaufmann, San Francisco (1995)Google Scholar
  30. 30.
    Castillo, E., Gutiérrez, J.M., Hadi, A.S.: Expert Systems and Probabilistic Network Models, Erste edn. Springer, New York (1996)Google Scholar
  31. 31.
    Aha, D.W., Kibler, D.: Instance-based learning algorithms. In: Machine Learning, pp. 37–66 (1991)Google Scholar
  32. 32.
    Witten, I.H., Frank, E.: Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations, 1st edn. The Morgan Kaufmann Series in Data Management Systems. Morgan Kaufmann, San Francisco (1999)Google Scholar
  33. 33.
    Tsymbal, A.: The problem of concept drift: Definitions and related work. Technical report, Department of Computer Science, Trinity College (2004)Google Scholar
  34. 34.
    Casson, A., Smith, S., Duncan, J., Rodriguez-Villegas, E.: Wearable EEG: what is it, why is it needed and what does it entail? In: Proc. IEEE Eng. Med. Biol. Soc., pp. 5867–5870 (2008)Google Scholar
  35. 35.
    Garipelli, G., Galán, F., Chavarriaga, R., Ferrez, P.W., Lew, E., Millán, J.: The use of Brain-Computer Interfacing for Ambient Intelligence. In: Intl. Workshop on Human Aspects in Ambient Intelligence (2007)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2010

Authors and Affiliations

  • Kilian Förster
    • 1
  • Andrea Biasiucci
    • 2
    • 3
  • Ricardo Chavarriaga
    • 2
  • José del R. Millán
    • 2
  • Daniel Roggen
    • 1
  • Gerhard Tröster
    • 1
  1. 1.IFE, Wearable Computing LabETH ZurichZurichSwitzerland
  2. 2.EPFL, CNBI, Center for NeuroprostheticsLausanneSwitzerland
  3. 3.Department of Informatics, Systems and Telematics (DIST)University of GenovaGenovaItaly

Personalised recommendations