Abstract
A challenging goal for cognitive neuroscience researchers is to determine how mental representations are mapped onto the patterns of neural activity. To address this problem, functional magnetic resonance imaging (fMRI) researchers have developed a large number of encoding and decoding methods. However, previous studies typically used rather limited stimuli representation, like semantic labels and Wavelet Gabor filters, and largely focused on voxel-based brain patterns. Here, we present a new fMRI encoding model to predict the human brain’s responses to free viewing of video clips which aims to deal with this limitation. In this model, we represent the stimuli using a variety of representative visual features in the computer vision community, which can describe the global color distribution, local shape and spatial information and motion information contained in videos, and apply the functional connectivity to model the brain’s activity pattern evoked by these video clips. Our experimental results demonstrate that brain network responses during free viewing of videos can be robustly and accurately predicted across subjects by using visual features. Our study suggests the feasibility of exploring cognitive neuroscience studies by computational image/video analysis and provides a novel concept of using the brain encoding as a test-bed for evaluating visual feature extraction.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Amir A, Berg M, Chang S-F, Hsu W, Iyengar G, Lin C-Y et al (2003) IBM research TRECVID-2003 video retrieval system. NIST TRECVID-2003
Baker S, Scharstein D, Lewis JP, Roth S, Black MJ, Szeliski R (2011) A database and evaluation methodology for optical flow. Int J Comput Vis 92(1):1–31
Bartels A, Zeki S (2004) Functional brain mapping during free viewing of natural scenes. Hum Brain Mapp 21(2):75–85
Bartels A, Zeki S, Logothetis NK (2008) Natural vision reveals regional specialization to local motion and to contrast-invariant, global flow in the human brain. Cereb Cortex 18(3):705–717
Chen M, Han J, Hu X, Jiang X, Guo L, Liu T (2014) Survey of encoding and decoding of visual stimulus via FMRI: an image analysis perspective. Brain Imaging Behav 8(1):7–23
Dumoulin SO, Wandell BA (2008) Population receptive field estimates in human visual cortex. Neuroimage 39(2):647–660
Friston KJ (2009) Modalities, modes, and models in functional neuroimaging. Science 326(5951):399
Friston KJ, Holmes AP, Poline JB, Grasby PJ, Williams SCR, Frackowiak RSJ et al (1995) Analysis of fMRI time-series revisited. Neuroimage 2(1):45–53
Hagmann P, Cammoun L, Gigandet X, Gerhard S, Ellen Grant P, Wedeen V et al (2010) MR connectomics: principles and challenges. J Neurosci Methods 194(1):34–45
Han J, Ji X, Hu X, Zhu D, Li K, Jiang X et al (2013) Representing and retrieving video shots in human-centric brain imaging space. IEEE Trans Image Process 22(7):2723–2736
Hasson U, Malach R, Heeger DJ (2010) Reliability of cortical activity during natural stimulation. Trends Cogn Sci 14(1):40–48
Haxby JV, Gobbini MI, Furey ML, Ishai A, Schouten JL, Pietrini P (2001) Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293(5539):2425–2430
Haynes J-D, Rees G (2006) Decoding mental states from brain activity in humans. Nat Rev Neurosci 7(7):523–534
Heeger DJ, Ress D (2002) What does fMRI tell us about neuronal activity? Nat Rev Neurosci 3(2):142–151
Hu X, Li K, Han J, Hua X, Guo L, Liu T (2012) Bridging the semantic gap via functional brain imaging. IEEE Trans Multimed 14(2):314–325
Kay KN, Gallant JL (2009) I can see what you see. Nat Neurosci 12(3):245
Kay KN, Naselaris T, Prenger RJ, Gallant JL (2008) Identifying natural images from human brain activity. Nature 452(7185):352–355
Laird AR, Eickhoff SB, Kurth F, Fox PM, Uecker AM, Turner JA et al. (2009). ALE meta-analysis workflows via the brainmap database: progress towards a probabilistic functional brain atlas. Front Neuroinform 3:23
Liu T (2011) A few thoughts on brain ROIs. Brain Imaging Behav 5(3):189–202
Logothetis NK, Pauls J, Augath M, Trinath T, Oeltermann A (2001) Neurophysiological investigation of the basis of the fMRI signal. Nature 412(6843):150–157
Lowe DG (2004) Distinctive image features from scale-invariant keypoints. Int J Comput Vis 60(2):91–110
Lynall M-E, Bassett DS, Kerwin R, McKenna PJ, Kitzbichler M, Muller U et al (2010) Functional connectivity and brain networks in schizophrenia. J Neurosci 30(28):9477–9487
Mitchell TM, Hutchinson R, Niculescu RS, Pereira F, Wang X, Just M et al (2004) Learning to decode cognitive states from brain images. Mach Learn 57(1–2):145–175
Mitchell TM, Shinkareva SV, Carlson A, Chang K-M, Malave VL, Mason RA et al (2008) Predicting human brain activity associated with the meanings of nouns. Science 320(5880):1191–1195
Miyawaki Y, Uchida H, Yamashita O, Sato M-A, Morito Y, Tanabe HC et al (2008) Visual image reconstruction from human brain activity using a combination of multiscale local image decoders. Neuron 60(5):915–929
Naselaris T, Prenger RJ, Kay KN, Oliver M, Gallant JL (2009) Bayesian reconstruction of natural images from human brain activity. Neuron 63(6):902–915
Naselaris T, Kay KN, Nishimoto S, Gallant JL (2011) Encoding and decoding in fMRI. Neuroimage 56(2):400–410
Nayak N, Sethi R, Song B, Roy-Chowdhury A (2011). Motion pattern analysis for modeling and recognition of complex human activities. Guide to Video Analysis of Humans: Looking at People. New York, Springer-Verlag, 289–310
Nishimoto S, Vu AT, Naselaris T, Benjamini Y, Yu B, Gallant JL (2011) Reconstructing visual experiences from brain activity evoked by natural movies. Curr Biol 21(19):1641–1646
Passingham RE, Stephan KE, Kotter R (2002) The anatomical basis of functional localization in the cortex. Nat Rev Neurosci 3(8):606–616
Peelen MV, Fei-Fei L, Kastner S (2009) Neural mechanisms of rapid natural scene categorization in human visual cortex. Nature 460(7251):94–97
Pereira F, Mitchell T, Botvinick M (2009) Machine learning classifiers and fMRI: a tutorial overview. Neuroimage 45(1):S199–S209
Polyn SM, Natu VS, Cohen JD, Norman KA (2005) Category-specific cortical activity precedes retrieval during memory search. Science 310(5756):1963–1966
Richiardi J, Eryilmaz H, Schwartz S, Vuilleumier P, Van De Ville D (2011) Decoding brain states from fMRI connectivity graphs. Neuroimage 56(2):616–626
Schneider W, Eschman A, Zuccolotto A (2002). E-Prime reference guide: Psychology Software Tools, Incorporated
Shirer WR, Ryali S, Rykhlevskaia E, Menon V, Greicius MD (2012) Decoding subject-driven cognitive states with whole-brain connectivity patterns. Cereb Cortex 22(1):158–165
Smeaton AF, Over P, Kraaij W (2006) Evaluation campaigns and TRECVid. In: Proceedings of the 8th ACM international workshop on multimedia information retrieval, ACM, pp 321–330
Sterzer P, Haynes, J-D, Rees G (2008). Fine-scale activity patterns in high-level visual areas encode the category of invisible objects. J Vis 8(15):10
Sugase-Miyamoto Y, Matsumoto N, Kawano K (2011) Role of temporal processing stages by inferior temporal neurons in facial recognition. Front Psychol 2:141
Suykens JAK, Vandewalle J (1999) Least squares support vector machine classifiers. Neural Process Lett 9(3):293–300
Thirion B, Pinel P, Mériaux S, Roche A, Dehaene S, Poline J-B (2007) Analysis of a large fMRI cohort: statistical and methodological issues for group analyses. Neuroimage 35(1):105–120
Van De Sande KEA, Gevers T, Snoek CGM (2010) Evaluating color descriptors for object and scene recognition. IEEE Trans Pattern Anal Mach Intell 32(9):1582–1596
Walther DB, Caddigan E, Fei-Fei L, Beck DM (2009) Natural scene categories revealed in distributed patterns of activity in the human brain. J Neurosci 29(34):10573–10581
Zhang T, Guo L, Li K, Jing C, Yin Y, Zhu D et al (2012) Predicting functional cortical ROIs via DTI-derived fiber shape models. Cereb Cortex 22(4):854–864
Zhu D, Li K, Faraco CC, Deng F, Zhang D, Guo L et al (2012) Optimization of functional brain ROIs via maximization of consistency of structural connectivity profiles. Neuroimage 59(2):1382–1393
Zhu D, Li K, Guo L, Jiang X, Zhang T, Zhang D et al (2013) DICCCOL: dense individualized and common connectivity-based cortical landmarks. Cereb Cortex 23(4):786–800
Acknowledgments
J Han was supported by the National Science Foundation of China under Grant 61005018 and 91120005, NPU-FFR-JC20120237 and Program for New Century Excellent Talents in University under Grant NCET-10-0079. X Hu was supported by the National Science Foundation of China under Grant 61103061 and Program for New Century Excellent Talents in University under grant NCET-13-0472. T Liu was supported by NIH Career Award (NIH EB 006878), NSF CAREER Award (IIS-1149260), NIH R01 DA033393, NSF BME-1302089 and NIH R01 AG-042599. L Guo was supported by the National Science Foundation of China under Grants 61273362 and 61333017.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Han, J., Zhao, S., Hu, X. et al. Encoding brain network response to free viewing of videos. Cogn Neurodyn 8, 389–397 (2014). https://doi.org/10.1007/s11571-014-9291-3
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11571-014-9291-3