Abstract
This article presents a novel, nonlinear, data-driven signal processing method, which can help neuroscience researchers visualize and understand complex dynamical patterns in both time and space. Specifically, we present applications of a Koopman operator approach for eigendecomposition of electrophysiological signals into orthogonal, coherent components and examine their associated spatiotemporal dynamics. This approach thus provides enhanced capabilities over conventional computational neuroscience tools restricted to analyzing signals in either the time or space domains. This is achieved via machine learning and kernel methods for data-driven approximation of skew-product dynamical systems. The approximations successfully converge to theoretical values in the limit of long embedding windows. First, we describe the method, then using electrocorticographic (ECoG) data from a mismatch negativity experiment, we extract time-separable frequencies without bandpass filtering or prior selection of wavelet features. Finally, we discuss in detail two of the extracted components, Beta (\( \sim \) 13 Hz) and high Gamma (\( \sim \) 50 Hz) frequencies, and explore the spatiotemporal dynamics of high- and low- frequency components.
Article PDF
Similar content being viewed by others
Explore related subjects
Find the latest articles, discoveries, and news in related topics.Avoid common mistakes on your manuscript.
References
Alho, K.: Cerebral generators of mismatch negativity (MMN) and its magnetic counterpart (MMNm) elicited by sound changes. Ear Hear. 16(1), 38–51 (1995)
Alho, K., Woods, D.L., Algazi, A., Knight, R.T., Näätänen, R.: Lesions of frontal cortex diminish the auditory mismatch negativity. Electroencephalogr. Clin. Neurophysiol. 91(5), 353–362 (1994). https://doi.org/10.1016/0013-4694(94)00173-1
Aru, J., Aru, J., Priesemann, V., Wibral, M., Lana, L., Pipa, G., Singer, W., Vicente, R.: Untangling cross-frequency coupling in neuroscience. Curr. Opin. Neurobiol. 31, 51–61 (2015). https://doi.org/10.1016/j.conb.2014.08.002
Aubry, N., Guyonnet, R., Lima, R.: Spatiotemporal analysis of complex signals: Theory and applications. J. Stat. Phys. 64, 683–739 (1991). https://doi.org/10.1007/bf01048312
Belkin, M., Niyogi, P.: Laplacian Eigenmaps for dimensionality reduction and data representation. Neural Comput. 15, 1373–1396 (2003). https://doi.org/10.1162/089976603321780317
Brunton, B.W., Johnson, L.A., Ojemann, J.G., Kutz, J.N.: Extracting spatial–temporal coherent patterns in large-scale neural recordings using dynamic mode decomposition. J. Neurosci. Methods 258, 1–15 (2016). https://doi.org/10.1016/j.jneumeth.2015.10.010
Budisić, M, Mohr, R, Mezić, I.: Applied Koopmanism. Chaos 22, 047510 (2012). https://doi.org/10.1063/1.4772195
Chandrasekaran, C., Turesson, H.K., Brown, C.H., Ghazanfar, A.A.: The Influence of Natural Scene Dynamics on Auditory Cortical Activity. J. Neurosci. 30(42), 13919–13931 (2010). https://doi.org/10.1523/JNEUROSCI.3174-10.2010
Coenen, A., Fine, E., Zayachkivska, O.A.: Beck: A forgotten pioneer in electroencephalography. Journal of the History of the Neurosciences: Basic and Clinical Perspectives 23(3), 276–286 (2014)
Coifman, R.R., Lafon, S.: Diffusion maps. Appl. Comput. Harmon. Anal. 21, 5–30 (2006). https://doi.org/10.1016/j.acha.2006.04.006
Cong, F., Kalyakin, I., Li, H., Huttunen-Scott, T., Huang, Y., Lyytinen, H., Ristaniemi, T.: Answering six questions in extracting children’s mismatch negativity through combining wavelet decomposition and independent component analysis. Cogn. Neurodyn. 5(4), 343–359 (2011). https://doi.org/10.1007/s11571-011-9161-1
Cong, F., Sipola, T., Huttunen-Scott, T., Xu, X., Ristaniemi, T., Lyytinen, H.: Hilbert-Huang versus Morlet wavelet transformation on mismatch negativity of children in uninterrupted sound paradigm. Nonlinear Biomedical Physics 3(1) (2009). https://doi.org/10.1186/1753-4631-3-1
Corlett, P.R., Marrouch, N.: Social cognitive neuroscience of attitudes and beliefs. In: Albarracín, D., Johnson, B. T. (eds.) Handbook of Attitudes and Attitude Change, vol. 1, pp 480–519. Taylor & Francis, New York (2018)
Csépe, V., Karmos, G., Molnár, M.: Evoked potential correlates of stimulus deviance during wakefulness and sleep in cat — animal model of mismatch negativity. Electroencephalogr. Clin. Neurophysiol. 66, 571–578 (1987). https://doi.org/10.1016/0013-4694(87)90103-9
Das, S., Giannakis, D.: Delay-coordinate maps and the spectra of Koopman operators. J. Stati. Phy. 14 (6), 1107–1145 (2019). https://doi.org/10.1007/s10955-019-02272-w
Dellnitz, M., Junge, O.: On the Approximation of Complicated Dynamical Behavior. SIAM J. Numer. Anal. 36, 491 (1999). https://doi.org/10.1137/S0036142996313002
van Drongelen, W.: Signal processing for neuroscientists. Elsevier, Amsterdam (2007)
Dürschmid, S., Edwards, E., Reichert, C., Dewar, C., Hinrichs, H., Heinze, H. -J., Kirsch, H.E., Dalal, S.S., Deouell, L.Y., Knight, R.T.: Hierarchy of prediction errors for auditory events in human temporal and frontal cortex. Proc. Natl. Acad. Sci. 113, 6755–6760 (2016). https://doi.org/10.1073/pnas.1525030113
Eisner, T., Farkas, B., Haase, M., Nagel, R.: Operator Theoretic Aspects of Ergodic Theory. Graduate Texts in Mathematics, vol. 272. Springer, Berlin (2015)
Escabí, M. A., Read, H.L., Viventi, J., Kim, D -H, Higgins, N.C., Storace, D.A., Liu, A.S.K., Gifford, A.M., Burke, J.A., Campisi, M., Kim, Y -S, Avrin, A.E., Van der Spiegel, J., Huang, Y., Li, M., Wu, J., Rogers, J.A., Litt, B., Cohen, Y.E.: A high-density, high-channel count, multiplexed μ ECoG array for auditory cortex. J Neurophysiol 112, 1566–1583 (2014). https://doi.org/10.1152/jn.00179.2013
Ford, J.M., Hillpard, S.A.: Event–related potentials (ERPs) to Interruptions of a Steady Rhythm. Psychophysiology 18(3), 322–330 (1981). https://doi.org/10.1111/j.1469-8986.1981.tb03043.x
Giannakis, D.: Dynamics-adapted cone kernels. SIAM J. Appl. Dyn. Sys. 14(2), 556–608 (2015). https://doi.org/10.1137/140954544
Giannakis, D.: Data-driven spectral decomposition and forecasting of ergodic dynamical systems. Appl. Comut. Harmon. Anal 47(2), 338–396 (2019). https://doi.org/10.1016/j.acha.2017.09.001
Giannakis, D., Das, S.: Extraction and prediction of coherent patterns in incompressible flows through space-time Koopman analysis. Phys. D. In revision. arXiv:1706.06450 (2017)
Giannakis, D., Kolchinskaya, A., Krasnov, D., Schumacher, J.: Koopman analysis of the long-term evolution in a turbulent convection cell. J. Fluid Mech. 847, 735–767 (2018). https://doi.org/10.1017/jfm.2018.297
Giannakis, D., Majda, A.J.: Time series reconstruction via machine learning: Revealing Decadal variability and intermittency in the North Pacific sector of a coupled climate model. In: Conference on Intelligent Data Understanding Proceedings, Mountain View, California (2011)
Giannakis, D., Majda, A.J.: Nonlinear Laplacian Spectral Analysis for Time Series with Intermittency and Low-Frequency Variability. Proc. Natl. Acad. Sci. 109 (7), 2222–2227 (2012). https://doi.org/10.1073/pnas.1118984109
Giannakis, D., Majda, A.J.: Nonlinear laplacian spectral analysis: capturing intermittent and low-frequency spatiotemporal patterns in high-dimensional data. Stat. Anal. Data Min. 6(3), 180–194 (2013). https://doi.org/10.1002/sam.11171
Giannakis, D., Ourmazd, A., Slawinska, J., Zhao, Z.: Spatiotemporal pattern extraction by spectral analysis of vector-valued observables. J. Nonlinear Sci., 1–61 (2019). https://doi.org/10.1007/s00332-019-09548-1
Giannakis, D., Slawinska, J.: Indo-pacific variability on seasonal to multidecadal time scales. Part II: Multiscale Atmosphere-Ocean Linkages. J. Climate 31, 693–725 (2018). https://doi.org/JCLI-D-17- 0031.1
Giannakis, D., Slawinska, J., Ourmazd, A., Zhao, Z.: Vector-valued spectral analysis of space-time data. In: Proceedings of the Time Series Workshop, Neural Information Processing Systems Conference, Long Beach, California (2017)
Giannakis, D., Slawinska, J., Zhao, Z.: Spatiotemporal feature extraction with data-driven Koopman operators. J. Mach. Learn. Res. Proceedings 44, 103–115 (2015)
Giard, M.H., Perrin, F., Pernier, J., Bouchet, P.: Brain generators implicated in the processing of auditory stimulus deviance: A topographic event-related potential study. Psychophysiology 27(6), 627–640 (1990). https://doi.org/10.1111/j.1469-8986.1990.tb03184.x
Ghil, M., et al.: Advanced spectral methods for climatic time series. Rev. Geophys. 40(1), 1003 (2002). https://doi.org/10.1029/2000RG000092
Grimm, S., Escera, C., Nelken, I.: Early indices of deviance detection in humans and animal models. Biol. Psychol. 116, 23–27 (2016). https://doi.org/10.1016/j.biopsycho.2015.11.017
Gumenyuk, V., Roth, T., Korzyukov, O., Jefferson, C., Kick, A., Spear, L., Tepley, N., Drake, C.L.: Shift work sleep disorder is associated with an attenuated brain response of sensory memory and an increased brain response to novelty: An ERP study. Sleep 33(5), 703–713 (2010)
Häenschel, C., Baldeweg, T., Croft, R.J., Whittington, M., Gruzelier, J.: Gamma and beta frequency oscillations in response to novel auditory stimuli: A comparison of human electroencephalogram (EEG) data with in vitro models. Proc. Natl. Acad. Sci. 97(13), 7645—7650 (2000). https://doi.org/10.1073/pnas.120162397
Hamilton, L.S., Edwards, E., Chang, E.F.: Parallel streams define the temporal dynamics of speech processing across human auditory cortex. bioRxiv 097485 (2017). https://doi.org/10.1101/097485
Harms, L., Michie, P.T., Näätänen, R.: Criteria for determining whether mismatch responses exist in animal models: Focus on rodents. Biol. Psychol. 116, 28–35 (2016). https://doi.org/10.1016/j.biopsycho.2015.07.006
Javitt, D.C.: Intracortical Mechanisms of Mismatch Negativity Dysfunction in Schizophrenia. Audiol. Neurotol. 5, 207–215 (2000). https://doi.org/10.1159/000013882
Javitt, D.C., Grochowski, S., Shelley, A.M., Ritter, W.: Impaired mismatch negativity (MMN) generation in schizophrenia as a function of stimulus deviance, probability, and interstimulus/interdeviant interval. Electroencephalogr. Clin. Neurophysiol. 108(2), 143–153 (1998). https://doi.org/10.1016/s0168-5597(97)00073-7
Komatsu, M., Takaura, K., Fujii, N.: Mismatch negativity in common marmosets: Whole-cortical recordings with multi-channel electrocorticograms. Sci. Rep. 5, 1005 (2017). https://doi.org/10.1038/srep15006
Koopman, B.O.: Hamiltonian systems and transformation in Hilbert space. Proc. Natl. Acad. Sci. 17(5), 315–318 (1931)
Lee, C.M., Osman, A.F., Volgushev, M., Escabí, M. A., Read, H.L.: Neural spike-timing patterns vary with sound shape and periodicity in three auditory cortical fields. J Neurophysiol 115, 1886–1904 (2016). https://doi.org/10.1152/jn.00784.2015
MacLean, S.E., Ward, L.M.: Temporo-frontal phase synchronization supports hierarchical network for mismatch negativity. Clin. Neurophysiol. 125(8), 1604–1617 (2014). https://doi.org/10.1016/j.clinph.2013.12.109
May, P.J.C., Tiitinen, H.: Mismatch negativity (MMN): The deviance-elicited auditory deflection explained. Psychophysiology 47(1), 66–122 (2010). https://doi.org/10.1111/j.1469-8986.2009.00856.x
Marrouch, N., Read, H.L., Slawinska, J., Giannakis, D.: Data-driven spectral decomposition of ECoG signal from an auditory oddball experiment in a marmoset monkey: implications for EEG data in humans. International Joint Conference on Neural Networks, 2161–4407 (2018)
Meindertsma, T., Kloosterman, N.A., Engel, A.K., Wagenmakers, E.J., Donner, T.H.: Surprise About Sensory Event Timing Drives Cortical Transients in the Beta Frequency Band. J. Neurosci. 38(35), 7600–7610 (2018). https://doi.org/10.1523/JNEUROSCI.0307-18.2018
Mezić, I.: Spectral properties of dynamical systems, model reduction and decompositions. Nonlinear Dyn. 41, 309–325 (2005). https://doi.org/10.1007/s11071-005-2824-x
Michie, P.T., Malmierca, M.S., Harms, L., Todd, J.: Understanding the neurobiology of MMN and its reduction in schizophrenia. Biol. Psychol. 116, 1–3 (2016). https://doi.org/10.1016/j.biopsycho.2016.02.005
Näätänen, R., Paavilainen, P., Alho, K., Reinikainen, K., Sams, M.: The mismatch negativity to intensity changes in an auditory stimulus sequence. Electroencephalogr. Clin. Neurophysiol. Suppl. 40, 125–131 (1987)
Näätänen, R., Paavilainen, P., Reinikainen, K.: Do event-related potentials to infrequent decrements in duration of auditory stimuli demonstrate a memory trace in man? Neurosci. Lett. 107(1-3), 347–352 (1989). https://doi.org/10.1016/0304-3940(89)90844-6
Näätänen, R., Todd, J., Schall, U.: Mismatch negativity (MMN) as biomarker predicting psychosis in clinically at-risk individuals. Biol. Psychol. 116, 36–40 (2016). https://doi.org/10.1016/j.biopsycho.2015.10.010
Rodgers, C.C., DeWeese, M.R.: Neural correlates of task switching in prefrontal cortex and primary auditory cortex in a novel stimulus selection task for rodents. Neuron 82(5), 1157–1170 (2014). https://doi.org/10.1016/j.neuron.2014.04.031
Ross, B., Schneider, B., Snyder, J.S., Alain, C.: Biological markers of auditory gap detection in young middle-aged and older adults. PLoS ONE 5(4), e10101 (2010). https://doi.org/10.1371/journal.pone.0010101
Schall, U.: Is it time to move mismatch negativity into the clinic? Biol. Psychol. 116, 41–46 (2016). https://doi.org/10.1016/j.biopsycho.2015.09.001
Slawinska, J., Ourmazd, A., Giannakis, D.: A quantum mechanical approach for data assimilation in climate dynamics. In: Workshop on “Climate Change: How can AI help?”, 36th International Conference on Machine Learning (ICML), Long Beach, California (2019). https://www.climatechange.ai/CameraReady/30/CameraReadySubmission/manuscript.pdf
Slawinska, J., Ourmazd, A., Giannakis, D.: A new approach to signal processing of spatiotemporal data. In: 2018 IEEE statistical signal processing workshop (SSP), pp. 338–342. https://doi.org/10.1109/SSP.2018.8450704 (2018)
Slawinska, J., Giannakis, D.: Indo-pacific variability on seasonal to multidecadal time scales. Part I: Intrinsic SST Modes in Models and Observations. J. Climate 30 (14), 5265–5294 (2017). https://doi.org/10.1175/jcli-d-16-0176.1
von Stein, A., Rappelsberger, P., Sarnthein, J., Petsche, H.: Synchronization between temporal and parietal cortex during multimodal object processing in man. Cereb. Cortex 9, 137—150 (1999). https://doi.org/10.1093/cercor/9.2.137
Tia, B., Takemi, M., Kosugi, A., Castagnola, E., Ansaldo, A., Nakamura, T., Ricci, D., Ushiba, J., Fadiga, L., Iriki, A.: Cortical control of object-specific grasp relies on adjustments of both activity and effective connectivity: A common marmoset study. J. Physiol. 595(6), 7203–7221 (2017). https://doi.org/10.1113/JP274629
Wacongne, C.: A predictive coding account of MMN reduction in schizophrenia. Biol. Psychol. 116, 68–74 (2016). https://doi.org/10.1016/j.biopsycho.2015.10.011
Widmann, A., Schröger, E.: Filter effects and filter artifacts in the analysis of electrophysiological data. Frontiers in psychology 3, 233 (2012). https://doi.org/10.3389/fpsyg.2012.00233
Acknowledgements
The authors would like to thank Ian Stevenson, Stephen Herzog, and the journal editorial team and reviewers for their helpful comments. Misako Komatsu, Kana Takura, and Naotaka Fujii from the RIKEN Brain Science Institute generously provided the open-access data used in this article. HLR has ownership interest in Elemind Technologies, Inc.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This article is an expanded version of research presented at the 2018 International Joint Conference on Neural Networks in Rio de Janeiro, Brazil. N. M. acknowledges support from the CT Institute for the Brain and Cognitive Sciences Graduate Summer Fellowship and the University of Connecticut Department of Psychological Sciences’ Maurice L. Farber Endowment. J. S. acknowledges support from NSF grants 1551489 and 1842538. D. G. acknowledges support from ONR YIP grant N00014-16-1-2649, NSF grant DMS-1521775, and DARPA grant HR0011-16-C-0116. H. R. acknowledges support from NSF grant 1355065, NIH DC015138 01, and the University of Connecticut Brain Computer Interface Core.
Rights and permissions
Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
About this article
Cite this article
Marrouch, N., Slawinska, J., Giannakis, D. et al. Data-driven Koopman operator approach for computational neuroscience. Ann Math Artif Intell 88, 1155–1173 (2020). https://doi.org/10.1007/s10472-019-09666-2
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10472-019-09666-2