Skip to main content

Survey of encoding and decoding of visual stimulus via FMRI: an image analysis perspective

Brain Imaging and Behavior volume 8, pages 7–23 (2014)Cite this article

Abstract

A variety of exciting scientific achievements have been made in the last few decades in brain encoding and decoding via functional magnetic resonance imaging (fMRI). This trend continues to rise in recent years, as evidenced by the increasing number of published papers in this topic and several published survey papers addressing different aspects of research issues. Essentially, these survey articles were mainly from cognitive neuroscience and neuroimaging perspectives, although computational challenges were briefly discussed. To complement existing survey articles, this paper focuses on the survey of the variety of image analysis methodologies, such as neuroimage registration, fMRI signal analysis, ROI (regions of interest) selection, machine learning algorithms, reproducibility analysis, structural and functional connectivity, and natural image analysis, which were employed in previous brain encoding/decoding research works. This paper also provides discussions of potential limitations of those image analysis methodologies and possible future improvements. It is hoped that extensive discussions of image analysis issues could contribute to the advancements of the increasingly important brain encoding/decoding field.

This is a preview of subscription content, access via your institution.

Access options

Buy single article

Instant access to the full article PDF.

43,34 €

Price includes VAT (Finland)
Tax calculation will be finalised during checkout.

References

  • Argyriou, A., Micchelli, C.A., Pontil, M., Ying, Y. (2008). A spectral regularization framework for multi-task structure learning. Advances in Neural Information Processing Systems: NIPS, pp. 25-32.

  • Arthurs, O. J., & Boniface, S. (2002). How well do we understand the neural origins of the fMRI BOLD signal? Trends in Neurosciences, 25, 27–31.

    CAS  PubMed  Google Scholar 

  • Avants, B. B., Epstein, C., Grossman, M., & Gee, J. C. (2008). Symmetric diffeomorphic image registration with cross-correlation: evaluating automated labeling of elderly and neurodegenerative brain. Medical Image Analysis, 12, 26–41.

    CAS  PubMed Central  PubMed  Google Scholar 

  • Bartels, A., & Zeki, S. (2004). Functional brain mapping during free viewing of natural scenes. Human Brain Mapping, 21, 75–85.

    PubMed  Google Scholar 

  • Bartels, A., & Zeki, S. (2005). Brain dynamics during natural viewing conditions—a new guide for mapping connectivity in vivo. NeuroImage, 24, 339–349.

    PubMed  Google Scholar 

  • Bartels, A., Zeki, S., & Logothetis, N. (2008). Natural vision reveals regional specialization to local motion and to contrast-invariant, global flow in the human brain. Cerebral Cortex, 18, 705–717.

    CAS  PubMed  Google Scholar 

  • Bashashati, A., Fatourechi, M., Ward, R. K., & Birch, G. E. (2007). A survey of signal processing algorithms in brain–computer interfaces based on electrical brain signals. Journal of Neural Engineering, 4, R32–R57.

    PubMed  Google Scholar 

  • Beauchamp, M. S., Lee, K. E., Haxby, J. V., & Martin, A. (2003). FMRI responses to video and point-light displays of moving humans and manipulable objects. Journal of Cognitive Neuroscience, 15, 991–1001.

    PubMed  Google Scholar 

  • Belliveau, J., Kennedy, D., Jr., McKinstry, R., Buchbinder, B., Weisskoff, R., Cohen, M., Vevea, J., Brady, T., & Rosen, B. (1991). Functional mapping of the human visual cortex by magnetic resonance imaging. Science, 254, 716–719.

    CAS  PubMed  Google Scholar 

  • Brouwer, G. J., & Heeger, D. J. (2009). Decoding and reconstructing color from responses in human visual cortex. Journal of Neuroscience, 29, 13992–14003.

    CAS  PubMed Central  PubMed  Google Scholar 

  • Carlson, T. A., Schrater, P., & He, S. (2003). Patterns of activity in the categorical representations of objects. Journal of Cognitive Neuroscience, 15, 704–717.

    PubMed  Google Scholar 

  • Chai, B., Walther, D., Beck, D., Li, F.-F. (2009). Exploring functional connectivities of the human brain using multivariate information analysis. Advances in Neural Information Processing Systems, pp. 270–278.

  • Chang, C., & Glover, G. H. (2010). Time-frequency dynamics of resting-state brain connectivity measured with fMRI. NeuroImage, 50, 81–98.

    PubMed Central  PubMed  Google Scholar 

  • Clithero, J. A., Smith, D. V., Carter, R. M., & Huettel, S. A. (2011). Within-and cross-participant classifiers reveal different neural coding of information. NeuroImage, 56, 699–708.

    PubMed Central  PubMed  Google Scholar 

  • Cox, D. D., & Savoy, R. L. (2003). Functional magnetic resonance imaging (fMRI)“brain reading”: detecting and classifying distributed patterns of fMRI activity in human visual cortex. NeuroImage, 19, 261–270.

    PubMed  Google Scholar 

  • Craddock, R. C., Holtzheimer, P. E., III, Hu, X. P., & Mayberg, H. S. (2009). Disease state prediction from resting state functional connectivity. Magnetic Resonance in Medicine, 62, 1619–1628.

    PubMed Central  PubMed  Google Scholar 

  • Davatzikos, C., Ruparel, K., Fan, Y., Shen, D., Acharyya, M., Loughead, J., Gur, R., & Langleben, D. D. (2005). Classifying spatial patterns of brain activity with machine learning methods: application to lie detection. Neuroethics Publications, 28, 663–668.

    CAS  Google Scholar 

  • Dayan, P., & Abbott, L. (2003). Theoretical neuroscience: computational and mathematical modeling of neural systems. Journal of Cognitive Neuroscience, 15, 154–155.

    Google Scholar 

  • Dayan, P., Abbott, L.F., Abbott, L. (2001). Theoretical neuroscience: computational and mathematical modeling of neural systems. Philosophical Psychology, pp. 563–577.

  • deCharms, R. C. (2008). Applications of real-time fMRI. Nature Reviews Neuroscience, 9, 720–729.

    CAS  PubMed  Google Scholar 

  • deCharms, R. C., & Merzenich, M. M. (1996). Primary cortical representation of sounds by the coordination of action-potential timing. Nature, 381, 610–613.

    CAS  PubMed  Google Scholar 

  • Deng, F., Zhu, D., Lv, J., Guo, L., Liu, T. (2013). FMRI signal analysis using empirical mean curve decomposition. IEEE transactions on biomedical engineering, 60, 42–54.

    Google Scholar 

  • Derrfuss, J., & Mar, R. A. (2009). Lost in localization: the need for a universal coordinate database. NeuroImage, 48, 1–7.

    PubMed  Google Scholar 

  • Dietterich, T. (1995). Overfitting and undercomputing in machine learning. ACM Computing Survey, 27, 326–327.

    Google Scholar 

  • Downing, P. E., Jiang, Y., Shuman, M., & Kanwisher, N. (2001). A cortical area selective for visual processing of the human body. Science, 293, 2470–2473.

    CAS  PubMed  Google Scholar 

  • Dumoulin, S. O., & Wandell, B. A. (2008). Population receptive field estimates in human visual cortex. NeuroImage, 39, 647–660.

    PubMed Central  PubMed  Google Scholar 

  • Eger, E., Ashburner, J., Haynes, J. D., Dolan, R. J., & Rees, G. (2008). fMRI activity patterns in human LOC carry information about object exemplars within category. Journal of Cognitive Neuroscience, 20, 356–370.

    PubMed Central  PubMed  Google Scholar 

  • Engel, S., Zhang, X., & Wandell, B. (1997). Colour tuning in human visual cortex measured with functional magnetic resonance imaging. Nature, 388, 68–71.

    CAS  PubMed  Google Scholar 

  • Engel, A. K., Fries, P., & Singer, W. (2001). Dynamic predictions: oscillations and synchrony in top-down processing. Nature Reviews Neuroscience, 2, 704–716.

    CAS  PubMed  Google Scholar 

  • Fischl, B., Salat, D. H., Busa, E., Albert, M., Dieterich, M., Haselgrove, C., van der Kouwe, A., Killiany, R., Kennedy, D., & Klaveness, S. (2002). Whole brain segmentation: automated labeling of neuroanatomical structures in the human brain. Neuron, 33, 341–355.

    CAS  PubMed  Google Scholar 

  • Fries, P. (2005). A mechanism for cognitive dynamics: neuronal communication through neuronal coherence. Trends in Cognitive Sciences, 9, 474–480.

    PubMed  Google Scholar 

  • Friston, K. J., Holmes, A. P., Worsley, K. J., Poline, J. P., Frith, C. D., & Frackowiak, R. S. J. (1994). Statistical parametric maps in functional imaging: a general linear approach. Human Brain Mapping, 2, 189–210.

    Google Scholar 

  • Friston, K. J., Holmes, A. P., Poline, J., Grasby, P., Williams, S., Frackowiak, R. S. J., & Turner, R. (1995). Analysis of fMRI time-series revisited. NeuroImage, 2, 45–53.

    CAS  PubMed  Google Scholar 

  • Friston, K. J., Holmes, A., Poline, J. B., Price, C. J., & Frith, C. (1996). Detecting activations in PET and fMRI: levels of inference and power. NeuroImage, 4, 223–235.

    CAS  PubMed  Google Scholar 

  • Friston, K., Chu, C., Mourão-Miranda, J., Hulme, O., Rees, G., Penny, W., & Ashburner, J. (2008). Bayesian decoding of brain images. NeuroImage, 39, 181–205.

    PubMed  Google Scholar 

  • Fujiwara, Y., Miyawaki, Y., Kamitani, Y. (2009). Estimating image bases for visual image reconstruction from human brain activity. Advances in Neural Information Processing Systems: NIPS, pp. 576–584.

  • Gao, W., & Lin, W. (2012). Frontal parietal control network regulates the anti-correlated default and dorsal attention networks. Human Brain Mapping, 33, 192–202.

    PubMed Central  PubMed  Google Scholar 

  • Gerstner, W., Kreiter, A. K., Markram, H., & Herz, A. V. M. (1997). Neural codes: firing rates and beyond. Proceedings of the National Academy of Sciences, 94, 12740–12741.

    CAS  Google Scholar 

  • Goferman, S., Zelnik-Manor, L., & Tal, A. (2012). Context-aware saliency detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34, 1915–1926.

    PubMed  Google Scholar 

  • Goffaux, V., Peters, J., Haubrechts, J., Schiltz, C., Jansma, B., & Goebel, R. (2011). From coarse to fine? Spatial and temporal dynamics of cortical face processing. Cerebral Cortex, 21, 467–476.

    PubMed  Google Scholar 

  • Golland, Y., Bentin, S., Gelbard, H., Benjamini, Y., Heller, R., Nir, Y., Hasson, U., & Malach, R. (2007). Extrinsic and intrinsic systems in the posterior cortex of the human brain revealed during natural sensory stimulation. Cerebral Cortex, 17, 766–777.

    PubMed  Google Scholar 

  • Hagmann, P., Cammoun, L., Gigandet, X., Gerhard, S., Ellen Grant, P., Wedeen, V., Meuli, R., Thiran, J. P., Honey, C. J., & Sporns, O. (2010). MR connectomics: principles and challenges. Journal of Neuroscience Methods, 194, 34–45.

    PubMed  Google Scholar 

  • Hasson, U., Nir, Y., Levy, I., Fuhrmann, G., & Malach, R. (2004). Intersubject synchronization of cortical activity during natural vision. Science, 303, 1634–1640.

    CAS  PubMed  Google Scholar 

  • Hasson, U., Landesman, O., Knappmeyer, B., Vallines, I., Rubin, N., & Heeger, D. J. (2008). Neurocinematics: the neuroscience of film. Projections, 2, 1–26.

    Google Scholar 

  • Hasson, U., Malach, R., & Heeger, D. J. (2010). Reliability of cortical activity during natural stimulation. Trends in Cognitive Sciences, 14, 40–48.

    PubMed Central  PubMed  Google Scholar 

  • Haxby, J. V., Gobbini, M. I., Furey, M. L., Ishai, A., Schouten, J. L., & Pietrini, P. (2001). Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science, 293, 2425–2430.

    CAS  PubMed  Google Scholar 

  • Haynes, J. D., & Rees, G. (2006). Decoding mental states from brain activity in humans. Nature Reviews Neuroscience, 7, 523–534.

    CAS  PubMed  Google Scholar 

  • Heeger, D. J., & Ress, D. (2002). What does fMRI tell us about neuronal activity? Nature Reviews Neuroscience, 3, 142–151.

    CAS  PubMed  Google Scholar 

  • Hu, X., Deng, F., Li, K., Zhang, T., Chen, H., Jiang, X., Lv, J., Zhu, D., Faraco, C., & Zhang, D. (2010). Bridging low-level features and high-level semantics via fMRI brain imaging for video classification. Proceedings of the International Conference on Multimedia: ICM (pp. 451–460). Firenze: ACM.

    Google Scholar 

  • Hu, X., Li, K., Han, J., Hua, X., Guo, L., & Liu, T. (2012). Bridging the semantic gap via functional brain imaging. IEEE Transactions on Multimedia, 14, 314–325.

    Google Scholar 

  • Hubel, D. H., & Wiesel, T. N. (1968). Receptive fields and functional architecture of monkey striate cortex. The Journal of Physiology, 195, 215–243.

    CAS  PubMed  Google Scholar 

  • Hubel, D. H., & Wiesel, T. N. (1969). Anatomical demonstration of columns in the monkey striate cortex. Nature, 221, 747–750.

    CAS  PubMed  Google Scholar 

  • Ishai, A., Ungerleider, L. G., Martin, A., Schouten, J. L., & Haxby, J. V. (1999). Distributed representation of objects in the human ventral visual pathway. Proceedings of the National Academy of Sciences, 96, 9379–9384.

    CAS  Google Scholar 

  • Ji, X., Han, J., Hu, X., Li, K., Deng, F., Fang, J., Guo, L., Liu, T. (2011). Retrieving video shots in semantic brain imaging space using manifold-ranking. International Conference on Image Processing: ICIP. IEEE, pp. 3633–3636.

  • Jiang, X., Zhang, T., Hu, X., Lu, L., Han, J., Guo, L., Liu, T. (2012). Music/speech classification using high-level features derived from fMRI brain imaging. Proceedings of the 20th ACM international conference on Multimedia. ACM, pp. 825–828.

  • Kamitani, Y., & Tong, F. (2005). Decoding the visual and subjective contents of the human brain. Nature Neuroscience, 8, 679–685.

    CAS  PubMed Central  PubMed  Google Scholar 

  • Kamitani, Y., & Tong, F. (2006). Decoding seen and attended motion directions from activity in the human visual cortex. Current Biology, 16, 1096–1102.

    CAS  PubMed Central  PubMed  Google Scholar 

  • Kanwisher, N., McDermott, J., & Chun, M. M. (1997). The fusiform face area: a module in human extrastriate cortex specialized for face perception. Journal of Neuroscience, 17, 4302–4311.

    CAS  PubMed  Google Scholar 

  • Kapoor, A., Shenoy, P., Tan, D., 2008. Combining brain computer interfaces with vision for object categorization. IEEE Conference on Computer Vision and Pattern Recognition: CVPR, pp. 1–8.

  • Kay, K. N., & Gallant, J. L. (2009). I can see what you see. Nature Neuroscience, 12, 245–245.

    CAS  PubMed  Google Scholar 

  • Kay, K. N., Naselaris, T., Prenger, R. J., & Gallant, J. L. (2008). Identifying natural images from human brain activity. Nature, 452, 352–355.

    CAS  PubMed Central  PubMed  Google Scholar 

  • Kennedy, D. N. (2010). Making connections in the connectome era. Neuroinformatics, 8, 61–62.

    PubMed  Google Scholar 

  • Kosslyn, S. M., Ganis, G., & Thompson, W. L. (2001). Neural foundations of imagery. Nature Reviews Neuroscience, 2, 635–642.

    CAS  PubMed  Google Scholar 

  • LaConte, S., Strother, S., Cherkassky, V., Anderson, J., & Hu, X. (2005). Support vector machines for temporal classification of block design fMRI data. NeuroImage, 26, 317–329.

    PubMed  Google Scholar 

  • LaConte, S. M., Peltier, S. J., & Hu, X. P. (2006). Real-time fMRI using brain-state classification. Human Brain Mapping, 28, 1033–1044.

    Google Scholar 

  • Laird, A. R., Eickhoff, S. B., Kurth, F., Fox, P. M., Uecker, A. M., Turner, J. A., Robinson, J. L., Lancaster, J. L., & Fox, P. T. (2009). ALE meta-analysis workflows via the BrainMap database: progress towards a probabilistic functional brain atlas. Frontiers in Neuroinformatics, 3, 1–11.

    Google Scholar 

  • Lebedev, M. A., & Nicolelis, M. A. L. (2006). Brain? machine interfaces: past, present and future. Trends in Neurosciences, 29, 536–546.

    CAS  PubMed  Google Scholar 

  • Lee, K., Tak, S., Ye, J.C. (2011). A data-driven sparse GLM for fMRI analysis using sparse dictionary learning with MDL criterion. IEEE Transactions on Medical Imaging, 30, 1076–1089.

    Google Scholar 

  • Li, J., Levine, M. D., An, X., & He, H. (2005). Saliency detection based on frequency and spatial domain analysis. Neuroscience, 8, 975–977.

    Google Scholar 

  • Li, K., Guo, L., Faraco, C., Zhu, D., Chen, H., Yuan, Y., Lv, J., Deng, F., Jiang, X., Zhang, T., Hu, X., Zhang, D., Miller, L. S., & Liu, T. (2012a). Visual analytics of brain networks. NeuroImage, 61, 82–97.

    CAS  PubMed  Google Scholar 

  • Li, K., Guo, L., Zhu, D., Hu, X., Han, J., & Liu, T. (2012b). Individual functional ROI optimization via maximization of group-wise consistency of structural and functional profiles. Neuroinformatics, 10, 225–242.

    CAS  PubMed  Google Scholar 

  • Li, K., Zhu, D., Guo, L., Li, Z., Lynch, M. E., Coles, C., Hu, X., & Liu, T. (2012c). Connectomics signatures of prenatal cocaine exposure affected adolescent brains. Human Brain Mapping. doi:10.1002/hbm.22082.

    Google Scholar 

  • Li, X., Lim, C., Li, K., Guo, L., & Liu, T. (2012d). Detecting Brain State Changes via Fiber-Centered Functional Connectivity Analysis. Neuroinformatics. doi:10.1007/s12021-012-9157-y.

    PubMed  Google Scholar 

  • Liu, T. (2011). A few thoughts on brain ROIs. Brain Imaging and Behavior, 5, 189–202.

    PubMed  Google Scholar 

  • Liu, Z., & He, B. (2008). fMRI-EEG integrated cortical source imaging by use of time-variant spatial constraints. NeuroImage, 39, 1198.

    CAS  PubMed Central  PubMed  Google Scholar 

  • Liu, T. M., Shen, D. G., & Davatzikos, C. (2003). Deformable registration of cortical structures via hybrid volumetric and surface warping. Medical Image Computing and Computer-Assisted Intervention: MICCAI, 2879, 780–787.

    Google Scholar 

  • Logothetis, N. K. (2008). What we can do and what we cannot do with fMRI. Nature, 453, 869–878.

    CAS  PubMed  Google Scholar 

  • Logothetis, N. K., Pauls, J., Augath, M., Trinath, T., & Oeltermann, A. (2001). Neurophysiological investigation of the basis of the fMRI signal. Nature, 412, 150–157.

    CAS  PubMed  Google Scholar 

  • Lowe, D. G. (2004). Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60, 91–110.

    Google Scholar 

  • MacEvoy, S. P., & Epstein, R. A. (2009). Decoding the representation of multiple simultaneous objects in human occipitotemporal cortex. Current Biology, 19, 943–947.

    CAS  PubMed Central  PubMed  Google Scholar 

  • Majeed, W., Magnuson, M., Hasenkamp, W., Schwarb, H., Schumacher, E. H., Barsalou, L., & Keilholz, S. D. (2011). Spatiotemporal dynamics of low frequency BOLD fluctuations in rats and humans. NeuroImage, 54, 1140–1150.

    PubMed Central  PubMed  Google Scholar 

  • Malinen, S., Hlushchuk, Y., & Hari, R. (2007). Towards natural stimulation in fMRI—issues of data analysis. NeuroImage, 35, 131–139.

    PubMed  Google Scholar 

  • Matthews, P., & Jezzard, P. (2004). Functional magnetic resonance imaging. Journal of Neurology, Neurosurgery, and Psychiatry, 75, 6–12.

    CAS  PubMed  Google Scholar 

  • Mechler, F., Victor, J. D., Purpura, K. P., & Shapley, R. (1998). Robust temporal coding of contrast by V1 neurons for transient but not for steady-state stimuli. Journal of Neuroscience, 18, 6583–6598.

    CAS  PubMed  Google Scholar 

  • Micchelli, C. A., Morales, J. M., & Pontil, M. (2010). A family of penalty functions for structured sparsity. Advances in Neural Information Processing Systems: NIPS, 23, 1612–1623.

    Google Scholar 

  • Mikl, M., Mareček, R., Hluštík, P., Pavlicová, M., Drastich, A., Chlebus, P., Brázdil, M., & Krupa, P. (2008). Effects of spatial smoothing on fMRI group inferences. Magnetic Resonance Imaging, 26, 490–503.

    PubMed  Google Scholar 

  • Mitchell, T. M., Hutchinson, R., Niculescu, R. S., Pereira, F., Wang, X., Just, M., & Newman, S. (2004). Learning to decode cognitive states from brain images. Machine Learning, 57, 145–175.

    Google Scholar 

  • Mitchell, T. M., Shinkareva, S. V., Carlson, A., Chang, K. M., Malave, V. L., Mason, R. A., & Just, M. A. (2008). Predicting human brain activity associated with the meanings of nouns. Science, 320, 1191–1195.

    CAS  PubMed  Google Scholar 

  • Miyawaki, Y., Uchida, H., Yamashita, O., Sato, M., Morito, Y., Tanabe, H. C., Sadato, N., & Kamitani, Y. (2008). Visual image reconstruction from human brain activity using a combination of multiscale local image decoders. Neuron, 60, 915–929.

    CAS  PubMed  Google Scholar 

  • Mourão-Miranda, J., Bokde, A. L., Born, C., Hampel, H., & Stetter, M. (2005). Classifying brain states and determining the discriminating activation patterns: Support Vector Machine on functional MRI data. NeuroImage, 28, 980–995.

    PubMed  Google Scholar 

  • Naselaris, T., Prenger, R. J., Kay, K. N., Oliver, M., & Gallant, J. L. (2009). Bayesian reconstruction of natural images from human brain activity. Neuron, 63, 902–915.

    CAS  PubMed  Google Scholar 

  • Naselaris, T., Kay, K. N., Nishimoto, S., & Gallant, J. L. (2011). Encoding and decoding in fMRI. NeuroImage, 56, 400–410.

    PubMed Central  PubMed  Google Scholar 

  • Naselaris, T., Stansbury, D. E., & Gallant, J. L. (2012). Cortical representation of animate and inanimate objects in complex natural scenes. Journal of Physiology, Paris, 106, 239–249.

    PubMed Central  PubMed  Google Scholar 

  • Nijholt, A., & Tan, D. (2008). Brain-computer interfacing for intelligent systems. Intelligent Systems, IEEE, 23, 72–79.

    Google Scholar 

  • Nishimoto, S., Vu, A. T., Naselaris, T., Benjamini, Y., Yu, B., & Gallant, J. L. (2011). Reconstructing visual experiences from brain activity evoked by natural movies. Current Biology, 21, 1641–1646.

    CAS  PubMed Central  PubMed  Google Scholar 

  • Norman, K. A., Polyn, S. M., Detre, G. J., & Haxby, J. V. (2006). Beyond mind-reading: multi-voxel pattern analysis of fMRI data. Trends in Cognitive Sciences, 10, 424–430.

    PubMed  Google Scholar 

  • O’Craven, K. M., & Kanwisher, N. (2000). Mental imagery of faces and places activates corresponding stimulus-specific brain regions. Journal of Cognitive Neuroscience, 12, 1013–1023.

    PubMed  Google Scholar 

  • Obozinski, G., Taskar, B., & Jordan, M. I. (2010). Joint covariate selection and joint subspace selection for multiple classification problems. Statistics and Computing, 20, 231–252.

    Google Scholar 

  • Ogawa, S., Lee, T., Kay, A., & Tank, D. (1990). Brain magnetic resonance imaging with contrast dependent on blood oxygenation. Proceedings of the National Academy of Sciences, 87, 9868–9872.

    CAS  Google Scholar 

  • Pantazatos, S. P., Talati, A., Pavlidis, P., & Hirsch, J. (2012). Decoding unattended fearful faces with whole-brain correlations: an approach to identify condition-dependent large-scale functional connectivity. PLoS Computational Biology, 8, e1002441.

    CAS  PubMed Central  PubMed  Google Scholar 

  • Passingham, R. E., Stephan, K. E., & Kotter, R. (2002). The anatomical basis of functional localization in the cortex. Nature Reviews Neuroscience, 3, 606–616.

    CAS  PubMed  Google Scholar 

  • Peelen, M. V., Fei-Fei, L., & Kastner, S. (2009). Neural mechanisms of rapid natural scene categorization in human visual cortex. Nature, 460, 94–97.

    CAS  PubMed Central  PubMed  Google Scholar 

  • Poldrack, R. A. (2006). Can cognitive processes be inferred from neuroimaging data? Trends in Cognitive Sciences, 10, 59–63.

    PubMed  Google Scholar 

  • Rasmussen, C. E., & Williams, C. (2006). Gaussian processes for machine learning (Vol. 38, pp. 715–719). Cambridge: The MIT Press.

    Google Scholar 

  • Rasmussen, P. M., Hansen, L. K., Madsen, K. H., Churchill, N. W., & Strother, S. C. (2012). Model sparsity and brain pattern interpretation of classification models in neuroimaging. Pattern Recognition, 45, 2085–2100.

    Google Scholar 

  • Redcay, E., Dodell-Feder, D., Pearrow, M. J., Mavros, P. L., Kleiner, M., Gabrieli, J. D., & Saxe, R. (2010). Live face-to-face interaction during fMRI: a new tool for social cognitive neuroscience. NeuroImage, 50, 1639–1647.

    PubMed Central  PubMed  Google Scholar 

  • Reddy, L., Tsuchiya, N., & Serre, T. (2010). Reading the mind’s eye: decoding category information during mental imagery. NeuroImage, 50, 818–825.

    PubMed Central  PubMed  Google Scholar 

  • Richiardi, J., Eryilmaz, H., Schwartz, S., Vuilleumier, P., & Van De Ville, D. (2011). Decoding brain states from fMRI connectivity graphs. NeuroImage, 56, 616–626.

    PubMed  Google Scholar 

  • Ryali, S., Supekar, K., Abrams, D.A., Menon, V. (2010). Sparse logistic regression for whole brain classification of fMRI data. Neuroimage, 51, 752–764.

    Google Scholar 

  • Sabuncu, M. R., Singer, B. D., Conroy, B., Bryan, R. E., Ramadge, P. J., & Haxby, J. V. (2010). Function-based intersubject alignment of human cortical anatomy. Cerebral Cortex, 20, 130–140.

    PubMed  Google Scholar 

  • Salek-Haddadi, A., Friston, K., Lemieux, L., & Fish, D. (2003). Studying spontaneous EEG activity with fMRI. Brain Research Reviews, 43, 110–133.

    CAS  PubMed  Google Scholar 

  • Schrouff, J., Phillips, C.L.M. (2012). Multivariate pattern recognition analysis: brain decoding. coma and disorders of consciousness, 35–43.

  • Sekiyama, K., Kanno, I., Miura, S., & Sugita, Y. (2003). Auditory-visual speech perception examined by fMRI and PET. Neuroscience Research, 47, 277–287.

    PubMed  Google Scholar 

  • Shen, D., & Davatzikos, C. (2002). HAMMER: hierarchical attribute matching mechanism for elastic registration. IEEE Transactions on Medical Imaging, 21, 1421–1439.

    PubMed  Google Scholar 

  • Shibata, K., Watanabe, T., Sasaki, Y., & Kawato, M. (2011). Perceptual learning incepted by decoded fMRI neurofeedback without stimulus presentation. Science, 334, 1413–1415.

    CAS  PubMed Central  PubMed  Google Scholar 

  • Shinkareva, S. V., Malave, V. L., Mason, R. A., Mitchell, T. M., & Just, M. A. (2011). Commonality of neural representations of words and pictures. NeuroImage, 54, 2418–2425.

    PubMed  Google Scholar 

  • Shirer, W., Ryali, S., Rykhlevskaia, E., Menon, V., & Greicius, M. (2012). Decoding subject-driven cognitive states with whole-brain connectivity patterns. Cerebral Cortex, 22, 158–165.

    CAS  PubMed  Google Scholar 

  • Singer, W. (1999). Neuronal synchrony: a versatile code review for the definition of relations? Neuron, 24, 49–65.

    CAS  PubMed  Google Scholar 

  • Sitaram, R., Caria, A., Veit, R., Gaber, T., Rota, G., Kuebler, A., & Birbaumer, N. (2007). fMRI brain-computer interface: a tool for neuroscientific research and treatment. Computational Intelligence and Neuroscience. doi:10.1155/2007/25487.

    PubMed Central  PubMed  Google Scholar 

  • Smith, S. M., Miller, K. L., Moeller, S., Xu, J., Auerbach, E. J., Woolrich, M. W., Beckmann, C. F., Jenkinson, M., Andersson, J., & Glasser, M. F. (2012). Temporally-independent functional modes of spontaneous brain activity. Proceedings of the National Academy of Sciences, 109, 3131–3136.

    CAS  Google Scholar 

  • Spreng, R. N., Stevens, W. D., Chamberlain, J. P., Gilmore, A. W., & Schacter, D. L. (2010). Default network activity, coupled with the frontoparietal control network, supports goal-directed cognition. NeuroImage, 53, 303–317.

    PubMed Central  PubMed  Google Scholar 

  • Sterzer, P., Haynes, J.-D., & Rees, G. (2008). Fine-scale activity patterns in high-level visual areas encode the category of invisible objects. Journal of Vision, 8, 1–12.

    PubMed  Google Scholar 

  • Stokes, M., Thompson, R., Cusack, R., & Duncan, J. (2009). Top-down activation of shape-specific population codes in visual cortex during mental imagery. Journal of Neuroscience, 29, 1565–1572.

    CAS  PubMed  Google Scholar 

  • Sugase-Miyamoto, Y., Matsumoto, N., & Kawano, K. (2011). Role of temporal processing stages by inferior temporal neurons in facial recognition. Frontiers in Psychology, 2, 1–8.

    Google Scholar 

  • Tahmasebi, A. (2010). Quantification of inter-subject variability in human brain and its impact on analysis of fMRI data. Kingston: School of Computing. Queen’s University.

    Google Scholar 

  • Tahmasebi, A. M., Abolmaesumi, P., Zheng, Z. Z., Munhall, K. G., & Johnsrude, I. S. (2009). Reducing inter-subject anatomical variation: Effect of normalization method on sensitivity of functional magnetic resonance imaging data analysis in auditory cortex and the superior temporal region. NeuroImage, 47, 1522–1531.

    PubMed Central  PubMed  Google Scholar 

  • Tenenbaum, J. B., De Silva, V., & Langford, J. C. (2000). A global geometric framework for nonlinear dimensionality reduction. Science, 290, 2319–2323.

    CAS  PubMed  Google Scholar 

  • Thirion, B., Duchesnay, E., Hubbard, E., Dubois, J., Poline, J.-B., Lebihan, D., & Dehaene, S. (2006). Inverse retinotopy: inferring the visual content of images from brain activation patterns. NeuroImage, 33, 1104–1116.

    PubMed  Google Scholar 

  • Thirion, B., Pinel, P., Mériaux, S., Roche, A., Dehaene, S., & Poline, J. B. (2007). Analysis of a large fMRI cohort: Statistical and methodological issues for group analyses. NeuroImage, 35, 105–120.

    PubMed  Google Scholar 

  • Thompson, P., & Toga, A. W. (1996). A surface-based technique for warping three-dimensional images of the brain. IEEE Transactions on Medical Imaging, 15, 402–417.

    CAS  PubMed  Google Scholar 

  • Trappenberg, T.P. (2010). Fundamentals of computational neuroscience. Oxford University Press.

  • Tsao, D. Y., Freiwald, W. A., Knutsen, T. A., Mandeville, J. B., & Tootell, R. B. H. (2003). Faces and objects in macaque cerebral cortex. Nature Neuroscience, 6, 989–995.

    CAS  PubMed  Google Scholar 

  • Tsao, D. Y., Freiwald, W. A., Tootell, R. B. H., & Livingstone, M. S. (2006). A cortical region consisting entirely of face-selective cells. Science, 311, 670–674.

    CAS  PubMed Central  PubMed  Google Scholar 

  • Van Dijk, K. R. A., Hedden, T., Venkataraman, A., Evans, K. C., Lazar, S. W., & Buckner, R. L. (2010). Intrinsic functional connectivity as a tool for human connectomics: theory, properties, and optimization. Journal of Neurophysiology, 103, 297–321.

    PubMed  Google Scholar 

  • Van Gerven, M., Farquhar, J., Schaefer, R., Vlek, R., Geuze, J., Nijholt, A., Ramsey, N., Haselager, P., Vuurpijl, L., & Gielen, S. (2009). The brain–computer interface cycle. Journal of Neural Engineering, 6, 041001.

    PubMed  Google Scholar 

  • van Gerven, M. A. J., de Lange, F. P., & Heskes, T. (2010). Neural decoding with hierarchical generative models. Neural Computation, 22, 3127–3142.

    PubMed  Google Scholar 

  • Velliste, M., Perel, S., Spalding, M. C., Whitford, A. S., & Schwartz, A. B. (2008). Cortical control of a prosthetic arm for self-feeding. Nature, 453, 1098–1101.

    CAS  PubMed  Google Scholar 

  • Villarreal, M. F., Fridman, E. A., & Leiguarda, R. C. (2012). The effect of the visual context in the recognition of symbolic gestures. PLoS One. doi:10.1371/journal.pone.0029644.

    Google Scholar 

  • Vu, V. Q., Ravikumar, P., Naselaris, T., Kay, K. N., Gallant, J. L., & Yu, B. (2011). Encoding and decoding V1 fMRI responses to natural images with sparse nonparametric models. Annals of Applied Statistics, 5, 1159–1182.

    PubMed Central  PubMed  Google Scholar 

  • Vulliemoz, S., Thornton, R., Rodionov, R., Carmichael, D., Guye, M., Lhatoo, S., McEvoy, A., Spinelli, L., Michel, C., & Duncan, J. (2009). The spatio-temporal mapping of epileptic networks: combination of EEG–fMRI and EEG source imaging. NeuroImage, 46, 834–843.

    CAS  PubMed Central  PubMed  Google Scholar 

  • Vulliemoz, S., Lemieux, L., Daunizeau, J., Michel, C. M., & Duncan, J. S. (2010). The combination of EEG source imaging and EEG–correlated functional MRI to map epileptic networks. Epilepsia, 51, 491–505.

    PubMed  Google Scholar 

  • Walther, D. B., Caddigan, E., Fei-Fei, L., & Beck, D. M. (2009). Natural scene categories revealed in distributed patterns of activity in the human brain. Journal of Neuroscience, 29, 10573–10581.

    CAS  PubMed Central  PubMed  Google Scholar 

  • Wang, J., Pohlmeyer, E., Hanna, B., Jiang, Y.-G., Sajda, P., Chang, S.-F. (2009). Brain state decoding for rapid image retrieval. Proceedings of the 17th ACM International Conference on Multimedia: ACMMM, 945―954.

  • Werner, S., & Noppeney, U. (2010). Distinct functional contributions of primary sensory and association areas to audiovisual integration in object categorization. Journal of Neuroscience, 30, 2662–2675.

    CAS  PubMed  Google Scholar 

  • Whittingstall, K., Bartels, A., Singh, V., Kwon, S., & Logothetis, N. K. (2010). Integration of EEG source imaging and fMRI during continuous viewing of natural movies. Magnetic Resonance Imaging, 28, 1135–1142.

    PubMed  Google Scholar 

  • Williams, R. (2010). The human connectome: just another’ome? Lancet Neurology, 9, 238–239.

    PubMed  Google Scholar 

  • Wolpaw, J. R., Birbaumer, N., McFarland, D. J., Pfurtscheller, G., & Vaughan, T. M. (2002). Brain-computer interfaces for communication and control. Clinical Neurophysiology, 113, 767–791.

    PubMed  Google Scholar 

  • Yacoub, E., Harel, N., & Uğurbil, K. (2008). High-field fMRI unveils orientation columns in humans. Proceedings of the National Academy of Sciences, 105, 10607–10612.

    CAS  Google Scholar 

  • Yamashita, O., Sato, M., Yoshioka, T., Tong, F., Kamitani, Y. (2008). Sparse estimation automatically selects voxels relevant for the decoding of fMRI activity patterns. Neuroimage, 424, 1414–1429.

    Google Scholar 

  • Yao, H., Shi, L., Han, F., Gao, H., & Dan, Y. (2007). Rapid learning in cortical coding of visual scenes. Nature Neuroscience, 10, 772–778.

    CAS  PubMed  Google Scholar 

  • Yue, Y., Loh, J. M., & Lindquist, M. A. (2010). Adaptive spatial smoothing of fMRI images. Statistics and its Interface, 3, 3–13.

    Google Scholar 

  • Zhang, P., Cootes, T., (2011). Automatic part selection for groupwise registration information processing in medical imaging. In: Székely, G., Hahn, H. (Eds.). Springer Berlin/Heidelberg, pp. 636-647.

  • Zhang, T., Guo, L., Li, K., Jing, C., Yin, Y., Zhu, D., Cui, G., Li, L., & Liu, T. (2012a). Predicting functional cortical ROIs via DTI-derived fiber shape models. Cerebral Cortex, 22, 854–864.

    PubMed  Google Scholar 

  • Zhang, X., Guo, L., Li, X., Zhu, D., Li, K., Sun, Z., Jin, C., Hu, X., Han, J., Zhao, Q. (2012b). Characterization of task-free/task-performance brain states. Medical Image Computing and Computer-Assisted Intervention–MICCAI 2012. Springer, pp. 237-245.

  • Zhu, D., Li, K., Faraco, C. C., Deng, F., Zhang, D., Guo, L., Miller, L. S., & Liu, T. (2012a). Optimization of functional brain ROIs via maximization of consistency of structural connectivity profiles. NeuroImage, 59, 1382–1393.

    PubMed Central  PubMed  Google Scholar 

  • Zhu, D., Li, K., Guo, L., Jiang, X., Zhang, T., Zhang, D., Chen, H., Deng, F., Faraco, C., Jin, C., Wee, C.-Y., Yuan, Y., Lv, P., Yin, Y., Hu, X., Duan, L., Hu, X., Han, J., Wang, L., Shen, D., Miller, L. S., Li, L., & Liu, T. (2012b). DICCCOL: Dense Individualized and Common Connectivity-Based Cortical Landmarks. Cerebral Cortex. doi:10.1093/cercor/bhs072.

    Google Scholar 

Download references

Acknowledgments

T. Liu was supported by the NIH Career Award (EB006878), NIH R01 HL087923-03S2, NIH R01 DA033393, NSF CAREER Award (IIS-1149260) and The University of Georgia start-up research funding. J. Han was supported by the National Science Foundation of China under Grant 61005018 and 91120005, NPU-FFR-JC20120237 and Program for New Century Excellent Talents in University under grant NCET-10-0079. L. Guo was supported by the National Science Foundation of China under grant 61273362. X. Hu was supported by the National Science Foundation of China under Grant 61103061, China Postdoctoral Science Foundation under Grant 20110490174 and 2012T50819. The author would like to thank the following collaborators for helpful discussions: Kaiming Li, Xiang Ji, Fan Deng, Dajiang Zhu, Tuo Zhang, Hanbo Chen, Xiang Li, Shu Zhang, Carlos, Faraco, L. Stephen Miller, Heng Huang, Xian-Sheng Hua, and Lie Lu.

Author information

Affiliations

  1. School of Automation, Northwestern Polytechnical University, Xi’an, China

    Mo Chen, Junwei Han, Xintao Hu & Lei Guo

  2. Cortical Architecture Imaging and Discovery Lab, Department of Computer Science and Bioimaging Research Center, The University of Georgia, Athens, GA, USA

    Xi Jiang & Tianming Liu

Authors
  1. Mo Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Junwei Han
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xintao Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Xi Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Lei Guo
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Tianming Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Tianming Liu.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Chen, M., Han, J., Hu, X. et al. Survey of encoding and decoding of visual stimulus via FMRI: an image analysis perspective. Brain Imaging and Behavior 8, 7–23 (2014). https://doi.org/10.1007/s11682-013-9238-z

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11682-013-9238-z

Keywords

  • fMRI
  • Encoding
  • Decoding
  • Vision
  • Visual stimulus