Abstract
Recent years have witnessed a rapid growth of interest in moving functional magnetic resonance imaging (fMRI) beyond simple scan-length averages and into approaches that can integrate structural MRI measures and capture rich multimodal interactions. It is becoming increasingly clear that multimodal fusion is able to provide more information for individual subjects by exploiting covariation between modalities, rather an analysis of each modality alone. Multimodal fusion is a more complicated endeavor that must be approached carefully and efficient methods should be developed to draw generalized and valid conclusions out of high dimensional data with a limited number of subjects, such as patients with brain disorders. Numerous research efforts have been reported in the field based on various statistical models, including independent component analysis (ICA), canonical correlation analysis (CCA), and partial least squares (PLS). In this chapter, we survey a number of methods previously shown in multimodal fusion reports, performed with or without prior information, and with their possible strengths and limitations addressed. To examine the function–structure associations of the brain in a more comprehensive and integrated manner, we also reviewed a number of multimodal studies that combined fMRI and structural (sMRI and/or diffusion tensor MRI) measures, which could reveal important brain alterations that may not be fully detected by employing separate analysis of individual modalities, and also enable us to identify potential brain illness biomarkers.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Skudlarski P, Jagannathan K, Calhoun VD et al (2008) Measuring brain connectivity: diffusion tensor imaging validates resting state temporal correlations. Neuroimage 43(3):554–561
Savopol F, Armenakis C (2002) Mergine of heterogeneous data for emergency mapping: data integration or data fusion? Proc. ISPRS
Ardnt C, Loffeld O (1996) Information gained by data fusion, Proc. SPIE, vol. 2784
Sui J, Adali T, Yu Q et al (2012) A review of multivariate methods for multimodal fusion of brain imaging data. J Neurosci Methods 204(1):68–81
Sui J, Huster R, Yu Q et al (2014) Function-structure associations of the brain: evidence from multimodal connectivity and covariance studies. Neuroimage 102P1:11–23
Schultz CC, Fusar-Poli P, Wagner G et al (2012) Multimodal functional and structural imaging investigations in psychosis research. Eur Arch Psychiatry Clin Neurosci 262(Suppl 2):S97–S106
Horwitz B, Poeppel D (2002) How can EEG/MEG and fMRI/PET data be combined? Hum Brain Mapp 17(1):1–3
Calhoun VD, Adali T (2009) Feature-based fusion of medical imaging data. IEEE Trans Inf Technol Biomed 13(5):711–720
Blum AL, Langley P (1997) Selection of relevant features and examples in machine learning. Artif Intell 97(1–2):245–271
Smith SM, Fox PT, Miller KL et al (2009) Correspondence of the brain’s functional architecture during activation and rest. Proc Natl Acad Sci U S A 106(31):13040–13045
Goldberg-Zimring D, Mewes AU, Maddah M et al (2005) Diffusion tensor magnetic resonance imaging in multiple sclerosis. J Neuroimaging 15(4 Suppl):68S–81S
Ramnani N, Lee L, Mechelli A et al (2002) Exploring brain connectivity: a new frontier in systems neuroscience. Functional Brain Connectivity, 4–6 April 2002, Dusseldorf, Germany. Trends Neurosci 25:496–497
Henson RN, Flandin G, Friston KJ et al (2010) A parametric empirical Bayesian framework for fMRI-constrained MEG/EEG source reconstruction. Hum Brain Mapp 31(10):1512–1531
Lemieux L (2004) Electroencephalography-correlated functional MR imaging studies of epileptic activity. Neuroimaging Clin N Am 14(3):487–506
Friston KJ (2009) Modalities, modes, and models in functional neuroimaging. Science 326(5951):399–403
De Martino F, Valente G, de Borst AW et al (2010) Multimodal imaging: an evaluation of univariate and multivariate methods for simultaneous EEG/fMRI. Magn Reson Imaging 28(8):1104–1112
Eichele T, Specht K, Moosmann M et al (2005) Assessing the spatiotemporal evolution of neuronal activation with single-trial event-related potentials and functional MRI. Proc Natl Acad Sci U S A 102(49):17798–17803
Astolfi L, Cincotti F, Mattia D et al (2004) Estimation of the effective and functional human cortical connectivity with structural equation modeling and directed transfer function applied to high-resolution EEG. Magn Reson Imaging 22(10):1457–1470
Rykhlevskaia E, Gratton G, Fabiani M (2008) Combining structural and functional neuroimaging data for studying brain connectivity: a review. Psychophysiology 45(2):173–187
Oakes TR, Fox AS, Johnstone T et al (2007) Integrating VBM into the General Linear Model with voxelwise anatomical covariates. Neuroimage 34(2):500–508
Schlosser R, Gesierich T, Kaufmann B et al (2003) Altered effective connectivity during working memory performance in schizophrenia: a study with fMRI and structural equation modeling. Neuroimage 19(3):751–763
Calhoun VD, Adali T, Kiehl KA et al (2006) A method for multitask fMRI data fusion applied to schizophrenia. Hum Brain Mapp 27(7):598–610
Correa N, Adali T, Calhoun VD (2007) Performance of blind source separation algorithms for fMRI analysis using a group ICA method. Magn Reson Imaging 25(5):684–694
Groves AR, Beckmann CF, Smith SM et al (2011) Linked independent component analysis for multimodal data fusion. Neuroimage 54(3):2198–2217
Sui J, He H, Pearlson GD et al (2012) Three-way (N-way) fusion of brain imaging data based on mCCA + jICA and its application to discriminating schizophrenia. Neuroimage 2(66):119–132
Sui J, Pearlson G, Caprihan A et al (2011) Discriminating schizophrenia and bipolar disorder by fusing fMRI and DTI in a multimodal CCA+ joint ICA model. Neuroimage 57(3):839–855
Chen K, Reiman EM, Huan Z et al (2009) Linking functional and structural brain images with multivariate network analyses: a novel application of the partial least square method. Neuroimage 47(2):602–610
Martinez-Montes E, Valdes-Sosa PA, Miwakeichi F et al (2004) Concurrent EEG/fMRI analysis by multiway Partial Least Squares. Neuroimage 22(3):1023–1034
Correa NM, Eichele T, Adali T et al (2010) Multi-set canonical correlation analysis for the fusion of concurrent single trial ERP and functional MRI. Neuroimage 50(4):1438–1445
Liu J, Pearlson G, Windemuth A et al (2009) Combining fMRI and SNP data to investigate connections between brain function and genetics using parallel ICA. Hum Brain Mapp 30(1):241–255
Sui J, Adali T, Pearlson GD et al (2009) A method for accurate group difference detection by constraining the mixing coefficients in an ICA framework. Hum Brain Mapp 30(9):2953–2970
Sui J, Adali T, Pearlson GD et al (2009) An ICA-based method for the identification of optimal FMRI features and components using combined group-discriminative techniques. Neuroimage 46(1):73–86
Caprihan A, Pearlson GD, Calhoun VD (2008) Application of principal component analysis to distinguish patients with schizophrenia from healthy controls based on fractional anisotropy measurements. Neuroimage 42(2):675–682
Liu J, Xu L, Calhoun VD (2008). Extracting principle components for discriminant analysis of FMRI images. ICASSP 449–452
Ulloa AE, Chen J, Vergara VM et al (2014) Association between copy number variation losses and alcohol dependence across African American and European American ethnic groups. Alcohol Clin Exp Res 38(5):1266–1274
Calhoun VD, Adali T, Giuliani NR et al (2006) Method for multimodal analysis of independent source differences in schizophrenia: combining gray matter structural and auditory oddball functional data. Hum Brain Mapp 27(1):47–62
Xu L, Pearlson G, Calhoun VD (2009) Joint source based morphometry identifies linked gray and white matter group differences. Neuroimage 44(3):777–789
Franco AR, Ling J, Caprihan A et al (2008) Multimodal and multi-tissue measures of connectivity revealed by joint independent component analysis. IEEE J Sel Top Signal Process 2(6):986–997
Calhoun VD, Adali T, Liu J (2006) A feature-based approach to combine functional MRI, structural MRI and EEG brain imaging data. Conf Proc IEEE Eng Med Biol Soc 1:3672–3675
Teipel SJ, Bokde AL, Meindl T et al (2010) White matter microstructure underlying default mode network connectivity in the human brain. Neuroimage 49(3):2021–2032
Eichele T, Calhoun VD, Debener S (2009) Mining EEG-fMRI using independent component analysis. Int J Psychophysiol 73(1):53–61
Xu L, Groth KM, Pearlson G et al (2009) Source-based morphometry: the use of independent component analysis to identify gray matter differences with application to schizophrenia. Hum Brain Mapp 30(3):711–724
Shenton ME, Dickey CC, Frumin M et al (2001) A review of MRI findings in schizophrenia. Schizophr Res 49(1–2):1–52
Correa NM, Li YO, Adali T et al (2008) Canonical correlation analysis for feature-based fusion of biomedical imaging modalities and its application to detection of associative networks in schizophrenia. IEEE J Sel Top Signal Process 2(6):998–1007
Li Y-O, Adali T, Wang W et al (2009) Joint blind source separation by multiset canonical correlation analysis. IEEE Trans Signal Process 57(10):3918–3929
Lin FH, McIntosh AR, Agnew JA et al (2003) Multivariate analysis of neuronal interactions in the generalized partial least squares framework: simulations and empirical studies. Neuroimage 20(2):625–642
Krishnan A, Williams LJ, McIntosh AR et al (2010) Partial Least Squares (PLS) methods for neuroimaging: a tutorial and review. Neuroimage 56(2):455–475
Correa NM, Adali T, Li YO et al (2010) Canonical correlation analysis for data fusion and group inferences: examining applications of medical imaging data. IEEE Signal Process Mag 27(4):39–50
McIntosh AR, Lobaugh NJ (2004) Partial least squares analysis of neuroimaging data: applications and advances. Neuroimage 23(Suppl 1):S250–S263
Camara E, Rodriguez-Fornells A, Munte TF (2010) Microstructural brain differences predict functional hemodynamic responses in a reward processing task. J Neurosci 30(34):11398–11402
Sui J, He H, Yu Q et al (2013) Combination of resting state fMRI, DTI, and sMRI data to discriminate schizophrenia by N-way MCCA + jICA. Front Hum Neurosci 7:235
Beckmann CF, Smith SM (2005) Tensorial extensions of independent component analysis for multisubject FMRI analysis. Neuroimage 25(1):294–311
Hyvarinen A, Oja E (2000) Independent component analysis: algorithms and applications. Neural Netw 13(4–5):411–430
Bell AJ, Sejnowski TJ (1995) An information-maximization approach to blind separation and blind deconvolution. Neural Comput 7(6):1129–1159
Bishop CM (1999) Variational principal components. Artif Neural Netw 7:509, Conference Publication No. 470
Liu J, Chen J, Ehrlich S et al (2014) Methylation patterns in whole blood correlate with symptoms in schizophrenia patients. Schizophr Bull 40(4):769–776
Segall JM, Allen EA, Jung RE et al (2012) Correspondence between structure and function in the human brain at rest. Front Neuroinform 6:10
Khullar S, Michael AM, Cahill ND et al (2011) ICA-fNORM: spatial normalization of fMRI data using intrinsic group-ICA networks. Front Syst Neurosci 5:93
Salgado-Pineda P, Junque C, Vendrell P et al (2004) Decreased cerebral activation during CPT performance: structural and functional deficits in schizophrenic patients. Neuroimage 21(3):840–847
Salgado-Pineda P, Fakra E, Delaveau P et al (2011) Correlated structural and functional brain abnormalities in the default mode network in schizophrenia patients. Schizophr Res 125(2–3):101–109
Skudlarski P, Jagannathan K, Anderson K et al (2010) Brain connectivity is not only lower but different in schizophrenia: a combined anatomical and functional approach. Biol Psychiatry 68(1):61–69
Schlosser RG, Nenadic I, Wagner G et al (2007) White matter abnormalities and brain activation in schizophrenia: a combined DTI and fMRI study. Schizophr Res 89(1–3):1–11
Staempfli P, Reischauer C, Jaermann T et al (2008) Combining fMRI and DTI: a framework for exploring the limits of fMRI-guided DTI fiber tracking and for verifying DTI-based fiber tractography results. Neuroimage 39(1):119–126
Koch K, Wagner G, Schachtzabel C et al (2011) Neural activation and radial diffusivity in schizophrenia: combined fMRI and diffusion tensor imaging study. Br J Psychiatry 198(3):223–229
Kim J, Lee JH (2012) Integration of structural and functional magnetic resonance imaging improves mild cognitive impairment detection. Magn Reson Imaging 31:718–732
Fan Y, Resnick SM, Wu X et al (2008) Structural and functional biomarkers of prodromal Alzheimer’s disease: a high-dimensional pattern classification study. Neuroimage 41(2):277–285
Wee CY, Yap PT, Zhang D et al (2011) Identification of MCI individuals using structural and functional connectivity networks. Neuroimage 59(3):2045–2056
Westlye LT, Walhovd KB, Bjornerud A et al (2009) Error-related negativity is mediated by fractional anisotropy in the posterior cingulate gyrus—a study combining diffusion tensor imaging and electrophysiology in healthy adults. Cereb Cortex 19(2):293–304
Sui J, Yu Q, He H et al (2012) A selective review of multimodal fusion methods in schizophrenia. Front Hum Neurosci 6:27
Zhang H, Liu L, Wu H, Fan Y (2012) Feature selection and SVM classification of multiple modality images for predicting MCI, in OHBM, Beijing, China.
Kim DI, Sui J, Rachakonda S et al (2010) Identification of imaging biomarkers in schizophrenia: a coefficient-constrained independent component analysis of the mind multi-site schizophrenia study. Neuroinformatics 8(4):213–229
Tian L, Meng C, Yan H et al (2011) Convergent evidence from multimodal imaging reveals amygdala abnormalities in schizophrenic patients and their first-degree relatives. PLoS One 6(12):e28794
Casey BJ, Tottenham N, Liston C et al (2005) Imaging the developing brain: what have we learned about cognitive development? Trends Cogn Sci 9(3):104–110
Smieskova R, Allen P, Simon A et al (2012) Different duration of at-risk mental state associated with neurofunctional abnormalities. A multimodal imaging study. Hum Brain Mapp 33(10):2281–2294
Rasser PE, Johnston P, Lagopoulos J et al (2005) Functional MRI BOLD response to Tower of London performance of first-episode schizophrenia patients using cortical pattern matching. Neuroimage 26(3):941–951
Fusar-Poli P, Broome MR, Woolley JB et al (2011) Altered brain function directly related to structural abnormalities in people at ultra high risk of psychosis: longitudinal VBM-fMRI study. J Psychiatr Res 45(2):190–198
Michael AM, Baum SA, White T et al (2010) Does function follow form?: methods to fuse structural and functional brain images show decreased linkage in schizophrenia. Neuroimage 49(3):2626–2637
Michael AM, King MD, Ehrlich S et al (2011) A data-driven investigation of gray matter-function correlations in schizophrenia during a working memory task. Front Hum Neurosci 5:71
Rektorova I, Mikl M, Barrett J et al (2012) Functional neuroanatomy of vocalization in patients with Parkinson’s disease. J Neurol Sci 313(1–2):7–12
Harms MP, Wang L, Csernansky JG et al (2012) Structure-function relationship of working memory activity with hippocampal and prefrontal cortex volumes. Brain Struct Funct 218(1):173–186
Choi K, Yang Z, Hu X et al (2008) A combined functional-structural connectivity analysis of major depression using joint independent components analysis. Psychiatric MRI/MRS 3555
Camchong J, MacDonald AW III, Bell C et al (2011) Altered functional and anatomical connectivity in schizophrenia. Schizophr Bull 37(3):640–650
Kim DJ, Park B, Park HJ (2012) Functional connectivity-based identification of subdivisions of the basal ganglia and thalamus using multilevel independent component analysis of resting state fMRI. Hum Brain Mapp 34:1371–1385
Olesen PJ, Nagy Z, Westerberg H et al (2003) Combined analysis of DTI and fMRI data reveals a joint maturation of white and grey matter in a fronto-parietal network. Brain Res Cogn Brain Res 18(1):48–57
Matthews SC, Strigo IA, Simmons AN et al (2011) A multimodal imaging study in U.S. veterans of Operations Iraqi and Enduring Freedom with and without major depression after blast-related concussion. Neuroimage 54(Suppl 1):S69–S75
Zhou Y, Shu N, Liu Y et al (2008) Altered resting-state functional connectivity and anatomical connectivity of hippocampus in schizophrenia. Schizophr Res 100(1–3):120–132
Yan H, Tian L, Yan J et al (2012) Functional and anatomical connectivity abnormalities in cognitive division of anterior cingulate cortex in schizophrenia. PLoS One 7(9):e45659
Soldner J, Meindl T, Koch W et al (2011) Structural and functional neuronal connectivity in Alzheimer’s disease: a combined DTI and fMRI study. Nervenarzt 83(7):878–887
Wang F, Kalmar JH, He Y et al (2009) Functional and structural connectivity between the perigenual anterior cingulate and amygdala in bipolar disorder. Biol Psychiatry 66(5):516–521
Schonberg T, Pianka P, Hendler T et al (2006) Characterization of displaced white matter by brain tumors using combined DTI and fMRI. Neuroimage 30(4):1100–1111
Voss HU, Schiff ND (2009) MRI of neuronal network structure, function, and plasticity. Prog Brain Res 175:483–496
Palacios EM, Sala-Llonch R, Junque C et al (2012) White matter integrity related to functional working memory networks in traumatic brain injury. Neurology 78(12):852–860
Jacobson S, Kelleher I, Harley M et al (2009) Structural and functional brain correlates of subclinical psychotic symptoms in 11–13 year old schoolchildren. Neuroimage 49(2):1875–1885
Supekar K, Uddin LQ, Prater K et al (2010) Development of functional and structural connectivity within the default mode network in young children. Neuroimage 52(1):290–301
Pomarol-Clotet E, Canales-Rodriguez EJ, Salvador R et al (2010) Medial prefrontal cortex pathology in schizophrenia as revealed by convergent findings from multimodal imaging. Mol Psychiatry 15(8):823–830
Sexton CE, Allan CL, Le Masurier M et al (2012) Magnetic resonance imaging in late-life depression: multimodal examination of network disruption. Arch Gen Psychiatry 69(7):680–689
Qiu MG, Ye Z, Li QY et al (2011) Changes of brain structure and function in ADHD children. Brain Topogr 24(3–4):243–252
Sui J, He H, Liu J et al (2012) Three-way FMRI-DTI-methylation data fusion based on mCCA + jICA and its application to schizophrenia. Conf Proc IEEE Eng Med Biol Soc 2012:2692–2695
Zhang D, Wang Y, Zhou L et al (2011) Multimodal classification of Alzheimer’s disease and mild cognitive impairment. Neuroimage 55(3):856–867
Groves AR, Smith SM, Fjell AM et al (2012) Benefits of multi-modal fusion analysis on a large-scale dataset: life-span patterns of inter-subject variability in cortical morphometry and white matter microstructure. Neuroimage 63(1):365–380
Sui J, Castro E, Hao H et al (2014) Combination of FMRI-SMRI-EEG data improves discrimination of schizophrenia patients by ensemble feature selection. The 36th Annual International Conference of the IEEE engineering in medicine and biology society (EMBC‘14) Chicago, Illinois, USA, no. August 26–30
Eichele T, Calhoun VD, Moosmann M et al (2008) Unmixing concurrent EEG-fMRI with parallel independent component analysis. Int J Psychophysiol 67(3):222–234
Haller S, Xekardaki A, Delaloye C et al (2011) Combined analysis of grey matter voxel-based morphometry and white matter tract-based spatial statistics in late-life bipolar disorder. J Psychiatry Neurosci 36(6):391–401
Chen Z, Cui L, Li M et al (2011) Voxel based morphometric and diffusion tensor imaging analysis in male bipolar patients with first-episode mania. Prog Neuropsychopharmacol Biol Psychiatry 36(2):231–238
Meda SA, Jagannathan K, Gelernter J et al (2010) A pilot multivariate parallel ICA study to investigate differential linkage between neural networks and genetic profiles in schizophrenia. Neuroimage 53(3):1007–1015
Jagannathan K, Calhoun VD, Gelernter J et al (2010) Genetic associations of brain structural networks in schizophrenia: a preliminary study. Biol Psychiatry 68(7):657–666
Jamadar S, Powers NR, Meda SA et al (2010) Genetic influences of cortical gray matter in language-related regions in healthy controls and schizophrenia. Schizophr Res 129:141–148
Hao X, Xu D, Bansal R et al (2011) Multimodal magnetic resonance imaging: the coordinated use of multiple, mutually informative probes to understand brain structure and function. Hum Brain Mapp 34(2):253–271
Fusar-Poli P, McGuire P, Borgwardt S (2011) Mapping prodromal psychosis: a critical review of neuroimaging studies. Eur Psychiatry 27(3):181–191
Meda SA, Gill A, Stevens MC et al (2012) Differences in resting-state functional magnetic resonance imaging functional network connectivity between schizophrenia and psychotic bipolar probands and their unaffected first-degree relatives. Biol Psychiatry 71(10):881–889
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2016 Springer Science+Business Media New York
About this protocol
Cite this protocol
Sui, J., Calhoun, V.D. (2016). Multimodal Fusion of Structural and Functional Brain Imaging Data. In: Filippi, M. (eds) fMRI Techniques and Protocols. Neuromethods, vol 119. Humana Press, New York, NY. https://doi.org/10.1007/978-1-4939-5611-1_28
Download citation
DOI: https://doi.org/10.1007/978-1-4939-5611-1_28
Published:
Publisher Name: Humana Press, New York, NY
Print ISBN: 978-1-4939-5609-8
Online ISBN: 978-1-4939-5611-1
eBook Packages: Springer Protocols