Skip to main content
SpringerLink
Log in

Decoding power-spectral profiles from FMRI brain activities during naturalistic auditory experience

  • Original Research
  • Published:
Brain Imaging and Behavior Aims and scope Submit manuscript

Abstract

Recent studies have demonstrated a close relationship between computational acoustic features and neural brain activities, and have largely advanced our understanding of auditory information processing in the human brain. Along this line, we proposed a multidisciplinary study to examine whether power spectral density (PSD) profiles can be decoded from brain activities during naturalistic auditory experience. The study was performed on a high resolution functional magnetic resonance imaging (fMRI) dataset acquired when participants freely listened to the audio-description of the movie “Forrest Gump”. Representative PSD profiles existing in the audio-movie were identified by clustering the audio samples according to their PSD descriptors. Support vector machine (SVM) classifiers were trained to differentiate the representative PSD profiles using corresponding fMRI brain activities. Based on PSD profile decoding, we explored how the neural decodability correlated to power intensity and frequency deviants. Our experimental results demonstrated that PSD profiles can be reliably decoded from brain activities. We also suggested a sigmoidal relationship between the neural decodability and power intensity deviants of PSD profiles. Our study in addition substantiates the feasibility and advantage of naturalistic paradigm for studying neural encoding of complex auditory information.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  • Abrams, D. A., Ryali, S., Chen, T., Chordia, P., Khouzam, A., Levitin, D. J., & Menon, V. (2013). Inter-subject synchronization of brain responses during natural music listening. European Journal of Neuroscience, 37(9), 1458–1469.

    Article  PubMed  PubMed Central  Google Scholar 

  • Alluri, V., Toiviainen, P., Jaaskelainen, I. P., Glerean, E., Sams, M., & Brattico, E. (2012). Large-scale brain networks emerge from dynamic processing of musical timbre, key and rhythm. NeuroImage, 59(4), 3677–3689.

    Article  PubMed  Google Scholar 

  • Alluri, V., Toiviainen, P., Lund, T. E., Wallentin, M., Vuust, P., Nandi, A. K., Ristaniemi, T., & Brattico, E. (2013). From Vivaldi to Beatles and back: predicting lateralized brain responses to music. NeuroImage, 83, 627–636.

    Article  PubMed  Google Scholar 

  • Angenstein, N., & Brechmann, A. (2015). Auditory intensity processing: categorization versus comparison. NeuroImage, 119, 362–370.

    Article  PubMed  Google Scholar 

  • Bartels, A., & Zeki, S. (2005). Brain dynamics during natural viewing conditions - a new guide for mapping connectivity in vivo. NeuroImage, 24(2), 339–349.

    Article  PubMed  Google Scholar 

  • Bilecen D, Seifritz E, Scheffler K, Henning J, AC S (2002) Amplitopicity of the human auditory cortex: an fMRI study. NeuroImage 17 (2):710–718.

  • Bordier, C., Puja, F., & Macaluso, E. (2013). Sensory processing during viewing of cinematographic material: computational modeling and functional neuroimaging. NeuroImage, 67, 213–226.

    Article  PubMed  Google Scholar 

  • Cong, F., Alluri, V., Nandi, A. K., Toiviainen, P., Rui, F., Abu-Jamous, B., Gong, L., Craenen, B. G. W., Poikonen, H., & Huotilainen, M. (2013). Linking brain responses to naturalistic music through analysis of ongoing EEG and stimulus features. Multimedia IEEE Transactions on, 15(5), 1060–1069.

    Article  Google Scholar 

  • Cortes, C., & Vapnik, V. (1995). Support-vector networks. Machine Learning, 20(3), 273–297.

    Google Scholar 

  • Dykstra, A. R., Koh, C. K., Braida, L. D., & Mark Jude, T. (2012). Dissociation of detection and discrimination of pure tones following bilateral lesions of auditory cortex. PloS One, 7(9), e44602.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Fang, J., Hu, X., Han, J., Jiang, X., Zhu, D., Guo, L., & Liu, T. (2015). Data-driven analysis of functional brain interactions during free listening to music and speech. Brain Imaging and Behavior, 9(2), 162–177.

    Article  PubMed  Google Scholar 

  • Farbood, M. M., Heeger, D. J., Marcus, G., Hasson, U., & Lerner, Y. (2015). The neural processing of hierarchical structure in music and speech at different timescales. Frontiers in Neuroscience, 9, 157.

    Article  PubMed  PubMed Central  Google Scholar 

  • Frey, B. J., & Dueck, D. (2007). Clustering by passing messages between data points. Science, 315(5814), 972–976.

    Article  CAS  PubMed  Google Scholar 

  • Han, J., Chen, C., Shao, L., Hu, X., Han, J., & Liu, T. (2015). Learning computational models of video memorability from FMRI brain imaging. IEEE Trans. On Cybernetics, 45(8), 1692–1703.

    Article  Google Scholar 

  • Hanke, M., Baumgartner, F. J., Ibe, P., Kaule, F. R., Pollmann, S., Speck, O., Zinke, W., & Stadler, J. (2014). A high-resolution 7-tesla fMRI dataset from complex natural stimulation with an audio movie. Scientific Data, 1, 140003.

    Article  PubMed  PubMed Central  Google Scholar 

  • Hasson, U., & Honey, C. (2012). Future trends in neuroimaging: neural processes as expressed within real-life contexts. NeuroImage, 62(2), 1272–1278.

    Article  PubMed  PubMed Central  Google Scholar 

  • Hasson, U., Nir, Y., Levy, I., Fuhrmann, G., & Malach, R. (2004). Intersubject synchronization of cortical activity during natural vision. Science, 303(5664), 1634–1640.

    Article  CAS  PubMed  Google Scholar 

  • Hinton, G. E. (2002). Training products of experts by minimizing contrastive divergence. Neural Computation, 14(8), 1771–1800.

    Article  PubMed  Google Scholar 

  • Hu, X., Lv, C., Cheng, G., Lv, J., Guo, L., Han, J., & Liu, T. (2015). Sparsity-constrained fMRI decoding of visual saliency in naturalistic video streams. Autonomous Mental Development, IEEE Transactions on 7, 2, 65–75.

    Google Scholar 

  • Huth, A. G., Nishimoto, S., Vu, A. T., & Gallant, J. L. (2012). A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron, 76(6), 1210–1224.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Jenkinson, M., Beckmann, C. F., Behrens, T. E., Woolrich, M. W., & Smith, S. M. (2012). FSL. NeuroImage, 62(2), 782–790.

    Article  PubMed  Google Scholar 

  • Ji, X., Han, J., Jiang, X., Hu, X., Guo, L., Han, J., Shao, L., & Liu, T. (2015). Analysis of music/speech via integration of audio content and functional brain response. Information Sciences, 297, 271–282.

    Article  Google Scholar 

  • Kauppi, J. P., Pajula, J., & Tohka, J. (2014). A versatile software package for inter-subject correlation based analyses of fMRI. Frontiers in Neuroinformatics, 8, 2.

    Article  PubMed  PubMed Central  Google Scholar 

  • Klein, M. E., & Zatorre, R. J. (2015). Representations of invariant musical categories are decodable by pattern analysis of locally distributed BOLD responses in superior temporal and intraparietal sulci. Cerebral Cortex, 25(7), 1947–1957.

    Article  PubMed  Google Scholar 

  • Kumar, S., Bonnici, H. M., Teki, S., Agus, T. R., Pressnitzer, D., Maguire, E. A., & TD, G. (2014). Representations of specific acoustic patterns in the auditory cortex and hippocampus. Proceedings Biological Sciences/The Royal Society, 281(1791), 20141000.

    Article  Google Scholar 

  • Langers, D. R., Van, D. P., Schoenmaker, E. S., & Backes, W. H. (2007). fMRI activation in relation to sound intensity and loudness. NeuroImage, 35(2), 709–718.

    Article  PubMed  Google Scholar 

  • Lasota, K., Ulmer, J., Firszt, J., Biswal, B., Daniels, D., & Prost, R. (2003). Intensity-dependent activation of the primary auditory cortex in functional magnetic resonance imaging. Journal of Computer Assisted Tomography, 27(2), 213–218.

    Article  PubMed  Google Scholar 

  • Lockwood, A., Salvi, R., Ml, A. S., Wack, D., Murphy, B., & Burkard, R. (1999). The functional anatomy of the normal human auditory system: responses to 0.5 and 4.0 kHz tones at varied intensities. Cerebral Cortex, 9(1), 65–76.

    Article  CAS  PubMed  Google Scholar 

  • Mohr, C. M., King, W. M., Freeman, A. J., Briggs, R. W., & Leonard, C. M. (1999). Influence of speech stimuli intensity on the activation of auditory cortex investigated with functional magnetic resonance imaging. Acoustical Society of America Journal, 105(5), 2738–2745.

    Article  CAS  Google Scholar 

  • Mustovic, H., Scheffler, K., Di Salle, F., Esposito, F., Neuhoff, J. G., Hennig, J., & Seifritz, E. (2003). Temporal integration of sequential auditory events: silent period in sound pattern activates human planum temporale. NeuroImage, 20(1), 429–434.

    Article  PubMed  Google Scholar 

  • Nardo, D., Santangelo, V., & Macaluso, E. (2011). Stimulus-driven orienting of visuo-spatial attention in complex dynamic environments. Neuron, 69(5), 1015–1028.

    Article  CAS  PubMed  Google Scholar 

  • Naselaris, T., Kay, K. N., Nishimoto, S., & Gallant, J. L. (2011). Encoding and decoding in fMRI. NeuroImage, 56(2), 400–410.

    Article  PubMed  Google Scholar 

  • Nishimoto, S., Vu, A. T., Naselaris, T., Benjamini, Y., Yu, B., & Gallant, J. L. (2011). Reconstructing visual experiences from brain activity evoked by natural movies. Current Biology, 21(19), 1641–1646.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Norman, K. A., Polyn, S. M., Detre, G. J., & Haxby, J. V. (2006). Beyond mind-reading: multi-voxel pattern analysis of fMRI data. Trends in Cognitive Sciences, 10(9), 424–430.

    Article  PubMed  Google Scholar 

  • Opitz, B., Rinne, T., Mecklinger, A., Von Cramon, D. Y., & Schröger, E. (2002). Differential contribution of frontal and temporal cortices to auditory change detection: fMRI and ERP results. NeuroImage, 15(1), 167–174.

    Article  PubMed  Google Scholar 

  • Proakis, J. G., & Manolakis, D. G. (1992). Digital signal processing: Principles, algorithms, and applications. Maxwell Macmillan Canada, Maxwell Macmillan International: Macmillan.

    Google Scholar 

  • Reiterer, S., Erb, M., Grodd, W., & Wildgruber, D. (2008). Cerebral processing of timbre and loudness: fMRI evidence for a contribution of Broca’s area to basic auditory discrimination. Brain Imaging and Behavior, 2(1), 1–10.

    Article  Google Scholar 

  • Röhl, M., & Uppenkamp, S. (2012). Neural coding of sound intensity and loudness in the human auditory system. Jaro, 13(3), 369–379.

    Article  PubMed  PubMed Central  Google Scholar 

  • Saenz, M., & Langers, D. (2014). Tonotopic mapping of human auditory cortex. Hearing Research, 307(1), 42–52.

    Article  PubMed  Google Scholar 

  • Santoro, R., Moerel, M., De, M. F., Goebel, R., Ugurbil, K., Yacoub, E., & Formisano, E. (2014). Encoding of natural sounds at multiple spectral and temporal resolutions in the human auditory cortex. PLoS Computational Biology, 10(1), e1003412.

    Article  PubMed  PubMed Central  Google Scholar 

  • Spiers, H. J., & Maguire, E. A. (2007). Decoding human brain activity during real-world experiences. Trends in Cognitive Sciences, 11(8), 356–365.

    Article  PubMed  Google Scholar 

  • Talavage, T. M., Sereno, M. I., Melcher, J. R., Ledden, P. J., Rosen, B. R., & Dale, A. M. (2004). Tonotopic organization in human auditory cortex revealed by progressions of frequency sensitivity. Journal of Neurophysiology, 91(3), 1282–1296.

    Article  PubMed  Google Scholar 

  • Toiviainen, P., Alluri, V., Brattico, E., Wallentin, M., & Vuust, P. (2013). Capturing the musical brain with lasso: dynamic decoding of musical features from fMRI data. NeuroImage, 88C, 170–180.

    Google Scholar 

  • Trost, W., Frühholz, S., Cochrane, T., Cojan, Y., & Vuilleumier, P. (2015). Temporal dynamics of musical emotions examined through intersubject synchrony of brain activity. Social Cognitive and Affective Neuroscience. doi:10.1093/scan/nsv060.

    PubMed  PubMed Central  Google Scholar 

  • Uppenkamp, S., & Röhl, M. (2013). Human auditory neuroimaging of intensity and loudness. Hearing Research, 307(1), 65–73.

    PubMed  Google Scholar 

  • Welch, P. D. (1967). The use of fast Fourier transform for the estimation of power spectra: a method based on time averaging over short, modified periodograms. IEEE Transactions on Audio and Electroacoustics, 15(2), 70–73.

    Article  Google Scholar 

  • Zhao S, Jiang X, Han J, Hu X, Zhu D, Lv J, Zhang T, Guo L, Liu T (2014) Decoding auditory saliency from FMRI brain imaging. Paper presented at the proceedings of the ACM international conference on multimedia, Orlando, Florida, USA.

Download references

Author information

Authors and Affiliations

  1. School of Automation, Northwestern Polytechnical University, Xi’an, China

    Xintao Hu, Lei Guo & Junwei Han

  2. Cortical Architecture Imaging and Discovery Lab, Department of Computer Science and Bioimaging Research Center, The University of Georgia, Athens, GA, USA

    Tianming Liu

Authors
  1. Xintao Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Lei Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Junwei Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Tianming Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Junwei Han.

Ethics declarations

Funding

This study was funded by National Natural Science Foundation of China (NSFC) 61103061, 61333017, 61473234 and 61522207, and the Fundamental Research Funds for the Central Universities 3102014JCQ01065.

Conflict of Interest

All co-authors have seen and agreed with the contents of the manuscript. We have no relevant conflicts of interest.

Ethical approval

This article does not contain any studies with human participants performed by any of the authors.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hu, X., Guo, L., Han, J. et al. Decoding power-spectral profiles from FMRI brain activities during naturalistic auditory experience. Brain Imaging and Behavior 11, 253–263 (2017). https://doi.org/10.1007/s11682-016-9515-8

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11682-016-9515-8

Keywords

Search

Navigation