Information Fusion for Perceptual Feedback: A Brain Activity Sonification Approach
When analysing multichannel processes, it is often convenient to use some sort of “visualisation” to help understand and interpret spatio-temporal dependencies between the channels, and to perform input variable selection. This is particularly advantageous when the levels of noise are high, the “active” channel changes its spatial location with time, and also for spatio-temporal processes where several channels contain meaningful information, such as in the case of electroencephalogram (EEG)-based brain activity monitoring. To provide insight into the dynamics of brain electrical responses, spatial sonification of multichannel EEG is performed, whereby the information from active channels is fused into music-like audio. Owing to its “data fusion via fission” mode of operation, empirical mode decomposition (EMD) is employed as a time-frequency analyser, and the brain responses to visual stimuli are sonified to provide audio feedback. Such perceptual feedback has enormous potential in multimodal brain computer and brain machine interfaces (BCI/BMI).
Unable to display preview. Download preview PDF.
- 1.Eerola, T., Toiviainen, P.: MIR in Matlab: The MIDI toolbox. In: Proceedings of the 5th International Conference on Music Information Retrieval, ISMIR2004, pp. 22–27. Audiovisual Institute, Universitat Pompeu Fabra, Barcelona, Spain (2004)Google Scholar
- 2.Fahle, M., Poggio, T. (eds.): Perceptual Learning. MIT, Cambridge, MA (2002)Google Scholar
- 3.Huang, N., Shen, Z., Long, S., Wu, M., Shih, H., Zheng, Q., Yen, N.C., Tung, C., Liu, H.: The empirical mode decomposition and the Hilbert spectrum for nonlinear and non-stationary time series analysis. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences 454(1971), 903–995 (1998)MATHCrossRefMathSciNetGoogle Scholar
- 7.Przybyszewski, A., Rutkowski, T.: Processing of the incomplete representation of the visual world. In: Proceedings of the First Warsaw International Seminar on Intelligent Systems, WISIS’04. Warsaw, Poland (2004)Google Scholar
- 8.Rilling, G., Flandrin, P., Goncalves, P.: On empirical mode decomposition and its algorithms. In: Proceedings of IEEE-EURASIP Workshop on Nonlinear Signal and Image Processing, NSIP-03. IEEE (2003)Google Scholar
- 9.Rutkowski, T.M., Vialatte, F., Cichocki, A., Mandic, D., Barros, A.K.: Knowledge-Based Intelligent Information and Engineering Systems, Lecture Notes in Artificial Intelligence, vol. 4253, chap. Auditory Feedback for Brain Computer Interface Management – An EEG Data Sonification Approach, pp. 1232–1239. Springer, Berlin, Heidelberg, New York (2006)Google Scholar