How to Integrate Data from Multiple Biological Layers in Mental Health?

  • Rogers F. Silva
  • Sergey M. PlisEmail author


Integrating information from multiple biological layers is a key approach to unraveling the complexities of the human brain, with its multiple overlapping structural and functional subsystems operating at widely different temporal and spatial scales. Moreover, it has true potential to positively impact mental health patients through early diagnosis and individualized treatment. This chapter lays out a succession of approaches to synergistic fusion of multimodal brain imaging data, with a special focus on blind source separation (BSS) and deep learning (DL) methods. Firstly, a broad unified description of the BSS field is introduced, serving as a theoretical backbone for the chapter. Complementary to that, a detailed case study of three different applications of joint independent component analysis (jICA) provides both a reference guide on data fusion and a bridge into more advanced BSS methods. Various advanced BSS methods such as multiset canonical correlation analysis (mCCA), multi-way partial least squares (N-PLS), independent vector analysis (IVA) and Parallel ICA are then reviewed and discussed in terms of their strengths and limitations. Finally, DL methods are introduced, focusing on three important applications: classification utilizing strategies for multimodal data augmentation, embedding of learned representations in order to reveal disease severity spectra, and multimodal tissue segmentation.


Data fusion Data reduction Deep learning Assignment matrix Biological layer Blind source separation Canonical correlation analysis Convolutional neural networks Independent component analysis Multidataset multidimensional 



We would like to thank Dr. Vince Calhoun for the useful discussions, as well as Alvaro Ulloa and Aleksandr Fedorov for kindly providing some of the images and results presented here. This work was supported by NIH grants R01EB006841 (SP), 2R01EB005846 (RS), and R01EB020407 (RS), NSF grants IIS-1318759 (SP), 1539067 (RS), and NIH NIGMS Center of Biomedical Research Excellent (COBRE) grant 5P20RR021938/P20GM103472/P30GM122734.


  1. Adalı T, Anderson M, Fu GS (2014) Diversity in independent component and vector analyses: Identifiability, algorithms, and applications in medical imaging. IEEE Signal Process Mag 31(3):18–33. CrossRefGoogle Scholar
  2. Adalı T, Levin-Schwartz Y, Calhoun VD (2015) Multimodal data fusion using source separation: Application to medical imaging. Proc IEEE 103(9):1494–1506. CrossRefGoogle Scholar
  3. Anderson M, Li XL, Adalı T (2010) Nonorthogonal independent vector analysis using multivariate gaussian model. In: Vigneron V, Zarzoso V, Moreau E, Gribonval R, Vincent E (eds) Proc LVA/ICA 2010, Lecture Notes in Computer Science, vol 6365. Springer, St. Malo, France, pp 354–361. Google Scholar
  4. Anderson M, Adalı T, Li XL (2012) Joint blind source separation with multivariate gaussian model: Algorithms and performance analysis. IEEE Trans Signal Process 60(4):1672–1683. CrossRefGoogle Scholar
  5. Anderson M, Fu GS, Phlypo R, Adalı T (2013) Independent vector analysis, the Kotz distribution, and performance bounds. In: Proc IEEE ICASSP 2013, Vancouver, BC, pp 3243–3247. Google Scholar
  6. Bell A, Sejnowski T (1995) An information-maximization approach to blind separation and blind deconvolution. Neural Comput 7(6):1129–1159PubMedCrossRefGoogle Scholar
  7. Belouchrani A, Abed-Meraim K, Cardoso JF, Moulines E (1993) Second-order blind separation of temporally correlated sources. In: Proc ICDSP 1993, Nicosia, Cyprus, pp 346–351Google Scholar
  8. Biessmann F, Plis S, Meinecke FC, Eichele T, Muller KR (2011) Analysis of multimodal neuroimaging data. IEEE Rev Biomed Eng 4:26–58. PubMedCrossRefGoogle Scholar
  9. Calhoun VD, Adalı T (2009) Feature-based fusion of medical imaging data. IEEE Trans Inf Technol Biomed 13(5):711–720. PubMedCrossRefGoogle Scholar
  10. Calhoun VD, Sui J (2016) Multimodal fusion of brain imaging data: A key to finding the missing link(s) in complex mental illness. Biol Psychiatry Cogn Neurosci Neuroimaging 1(3):230–244. PubMedPubMedCentralCrossRefGoogle Scholar
  11. Calhoun VD, Adalı T, Giuliani NR, Pekar JJ, Kiehl KA, Pearlson GD (2006a) Method for multimodal analysis of independent source differences in schizophrenia: Combining gray matter structural and auditory oddball functional data. Hum Brain Mapp 27(1):47–62. PubMedCrossRefGoogle Scholar
  12. Calhoun VD, Adalı T, Kiehl K, Astur R, Pekar J, Pearlson G (2006b) A method for multi-task fMRI data fusion applied to schizophrenia. Hum Brain Mapp 27(7):598–610. PubMedPubMedCentralCrossRefGoogle Scholar
  13. Calhoun VD, Adalı T, Pearlson GD, Kiehl KA (2006c) Neuronal chronometry of target detection: Fusion of hemodynamic and event-related potential data. NeuroImage 30(2):544–553. PubMedCrossRefGoogle Scholar
  14. Cardoso JF (1998) Multidimensional independent component analysis. In: Proc IEEE ICASSP 1998, Seattle, WA, vol 4, pp 1941–1944.
  15. Castro E, Ulloa A, Plis SM, Turner JA, Calhoun VD (2015) Generation of synthetic structural magnetic resonance images for deep learning pre-training. In: Proc IEEE ISBI 2015, pp 1057–1060. Google Scholar
  16. Chen K, Reiman EM, Huan Z, Caselli RJ, Bandy D, Ayutyanont N, Alexander GE (2009) Linking functional and structural brain images with multivariate network analyses: A novel application of the partial least square method. NeuroImage 47(2):602–610. PubMedPubMedCentralCrossRefGoogle Scholar
  17. Çiçek Ö, Abdulkadir A, Lienkamp SS, Brox T, Ronneberger O (2016) 3D U-Net: learning dense volumetric segmentation from sparse annotation. In: Proc MICCAI 2016, pp 424–432. Google Scholar
  18. Comon P (1994) Independent component analysis, a new concept? Signal Process 36(3):287–314. CrossRefGoogle Scholar
  19. Comon P, Jutten C (2010) Handbook of blind source separation, 1st edn. Academic Press, Oxford, UKGoogle Scholar
  20. Correa NM, Li YO, Adalı T, Calhoun VD (2008) Canonical correlation analysis for feature-based fusion of biomedical imaging modalities and its application to detection of associative networks in schizophrenia. IEEE J Sel Topics Signal Process 2(6):998–1007. CrossRefGoogle Scholar
  21. Correa NM, Li YO, Adalı T, Calhoun VD (2009) Fusion of fMRI, sMRI, and EEG data using canonical correlation analysis. In: Proc IEEE ICASSP 2009, pp 385–388. Google Scholar
  22. Correa NM, Eichele T, Adalı T, Li YO, Calhoun VD (2010) Multi-set canonical correlation analysis for the fusion of concurrent single trial ERP and functional MRI. Neuroimage 50(4):1438–1445. PubMedPubMedCentralCrossRefGoogle Scholar
  23. Dähne S, Bießmann F, Meinecke F, Mehnert J, Fazli S, Müller KR (2013) Integration of multivariate data streams with bandpower signals. IEEE Trans Multimedia 15(5):1001–1013. CrossRefGoogle Scholar
  24. Dähne S, Meinecke F, Haufe S, Höhne J, Tangermann M, Müller KR, Nikulin V (2014a) SPoC: A novel framework for relating the amplitude of neuronal oscillations to behaviorally relevant parameters. NeuroImage 86:111–122. PubMedCrossRefGoogle Scholar
  25. Dähne S, Nikulin V, Ramírez D, Schreier P, Müller KR, Haufe S (2014b) Finding brain oscillations with power dependencies in neuroimaging data. NeuroImage 96:334–348. PubMedCrossRefGoogle Scholar
  26. Dale AM, Fischl B, Sereno MI (1999) Cortical surface-based analysis: I. segmentation and surface reconstruction. NeuroImage 9(2):179–194. PubMedCrossRefGoogle Scholar
  27. Fedorov A, Damaraju E, Calhoun V, Plis S (2017a) Almost instant brain atlas segmentation for large-scale studies. arXiv preprint URL
  28. Fedorov A, Johnson J, Damaraju E, Ozerin A, Calhoun V, Plis S (2017b) End-to-end learning of brain tissue segmentation from imperfect labeling. In: Proc IJCNN 2017, pp 3785–3792. Google Scholar
  29. Goodfellow I, Bengio Y, Courville A (2016) Deep learning. MIT Press.
  30. Haykin S (2008) Neural networks and learning machines, 3rd edn. Prentice Hall, Upper Saddle River, NJGoogle Scholar
  31. Hotelling H (1936) Relations between two sets of variates. Biometrika 28(3/4):321–377. CrossRefGoogle Scholar
  32. Hyvärinen A, Erkki O (1997) A fast fixed-point algorithm for independent component analysis. Neural Comput 9(7):1483–1492. CrossRefGoogle Scholar
  33. Hyvärinen A, Köster U (2006) FastISA: A fast fixed-point algorithm for independent subspace analysis. In: Proc ESANN 2006, Bruges, Belgium, pp 371–376Google Scholar
  34. Hyvärinen A, Karhunen J, Oja E (2002) Independent component analysis, 1st edn. Wiley, New York, NYGoogle Scholar
  35. Ioffe S, Szegedy C (2015) Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: Proc ICML 2015, Lille, France, vol 37, pp 448–456Google Scholar
  36. Karahan E, Rojas-López PA, Bringas-Vega ML, Valdés-Hernández PA, Valdés-Sosa PA (2015) Tensor analysis and fusion of multimodal brain images. Proc IEEE 103(9):1531–1559. CrossRefGoogle Scholar
  37. Kettenring J (1971) Canonical analysis of several sets of variables. Biometrika 58(3):433–451. CrossRefGoogle Scholar
  38. Kim T, Eltoft T, Lee TW (2006) Independent vector analysis: An extension of ICA to multivariate components. In: Proc ICA 2006, Springer, Charleston, SC, Lecture Notes in Computer Science, vol 3889, pp 165–172. CrossRefGoogle Scholar
  39. Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Proc NIPS 2012, pp 1097–1105Google Scholar
  40. Lahat D, Jutten C (2015) Joint independent subspace analysis: A quasi-Newton algorithm. In: Proc LVA/ICA 2015, Springer, Liberec, Czech Republic, Lecture Notes in Computer Science, vol 9237, pp 111–118. CrossRefGoogle Scholar
  41. Lahat D, Cardoso J, Messer H (2012) Second-order multidimensional ICA: Performance analysis. IEEE Trans Signal Process 60(9):4598–4610. CrossRefGoogle Scholar
  42. Lahat D, Adalı T, Jutten C (2015) Multimodal data fusion: An overview of methods, challenges, and prospects. Proc IEEE 103(9):1449–1477. CrossRefGoogle Scholar
  43. Liu J, Pearlson G, Calhoun V, Windemuth A (2007) A novel approach to analyzing fMRI and SNP data via parallel independent component analysis. Proc SPIE 6511:651,113–651,113–11.
  44. Liu J, Pearlson G, Windemuth A, Ruano G, Perrone-Bizzozero NI, Calhoun VD (2009) Combining fMRI and SNP data to investigate connections between brain function and genetics using parallel ICA. Hum Brain Mapp 30(1):241–255. PubMedPubMedCentralCrossRefGoogle Scholar
  45. Long J, Shelhamer E, Darrell T (2015) Fully convolutional networks for semantic segmentation. In: Proc IEEE CVPR 2015, pp 3431–3440. Google Scholar
  46. Lorenzi M, Simpson IJ, Mendelson AF, Vos SB, Cardoso MJ, Modat M, Schott JM, Ourselin S (2016) Multimodal image analysis in Alzheimer’s disease via statistical modelling of non-local intensity correlations. Sci Rep 6:22,161. CrossRefGoogle Scholar
  47. Maaten Lvd, Hinton G (2008) Visualizing data using t-SNE. J Mach Learn Res 9(Nov):2579–2605Google Scholar
  48. Martínez-Montes E, Valdés-Sosa PA, Miwakeichi F, Goldman RI, Cohen MS (2004) Concurrent EEG/fMRI analysis by multiway partial least squares. NeuroImage 22(3):1023–1034. PubMedCrossRefGoogle Scholar
  49. Meda S, Narayanan B, Liu J, Perrone-Bizzozero N, Stevens M, Calhoun VD, Glahn D, Shen L, Risacher S, Saykin A, Pearlson G (2012) A large scale multivariate parallel ICA method reveals novel imaging-genetic relationships for Alzheimer’s disease in the ADNI cohort. NeuroImage 60(3):1608–1621. PubMedPubMedCentralCrossRefGoogle Scholar
  50. Meier T, Wildenberg J, Liu J, Chen J, Calhoun VD, Biswal B, Meyerand M, Birn R, Prabhakaran V (2012) Parallel ICA identifies sub-components of resting state networks that covary with behavioral indices. Front Hum Neurosci 6:281. PubMedPubMedCentralCrossRefGoogle Scholar
  51. Miller KL, Alfaro-Almagro F, Bangerter NK, Thomas DL, Yacoub E, Xu J, Bartsch AJ, Jbabdi S, Sotiropoulos SN, Andersson JLR, Griffanti L, Douaud G, Okell TW, Weale P, Dragonu I, Garratt S, Hudson S, Collins R, Jenkinson M, Matthews PM, Smith SM (2016) Multimodal population brain imaging in the UK Biobank prospective epidemiological study. Nat Neurosci 19(11):1523–1536. PubMedPubMedCentralCrossRefGoogle Scholar
  52. Mohammadi-Nejad AR, Hossein-Zadeh GA, Soltanian-Zadeh H (2017) Structured and sparse canonical correlation analysis as a brain-wide multi-modal data fusion approach. IEEE Trans Med Imaging 36(7):1438–1448. PubMedCrossRefGoogle Scholar
  53. Plis SM, Hjelm DR, Salakhutdinov R, Allen EA, Bockholt HJ, Long JD, Johnson HJ, Paulsen JS, Turner JA, Calhoun VD (2014) Deep learning for neuroimaging: a validation study. Front Neurosci 8:229. PubMedPubMedCentralCrossRefGoogle Scholar
  54. Silva RF, Plis SM, Adalı T, Calhoun VD (2014a) Multidataset independent subspace analysis. In: Proc OHBM 2014, Poster 3506Google Scholar
  55. Silva RF, Plis SM, Adalı T, Calhoun VD (2014b) Multidataset independent subspace analysis extends independent vector analysis. In: Proc IEEE ICIP 2014, Paris, France, pp 2864–2868. Google Scholar
  56. Silva RF, Plis SM, Adalı T, Calhoun VD (2014c) A statistically motivated framework for simulation of stochastic data fusion models applied to multimodal neuroimaging. NeuroImage 102, Part 1:92–117. Google Scholar
  57. Silva RF, Plis SM, Sui J, Pattichis MS, Adalı T, Calhoun VD (2016) Blind source separation for unimodal and multimodal brain networks: A unifying framework for subspace modeling. IEEE J Sel Topics Signal Process 10(7):1134–1149. CrossRefGoogle Scholar
  58. Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R (2014) Dropout: A simple way to prevent neural networks from overfitting. J Mach Learn Res 15:1929–1958Google Scholar
  59. Sui J, Adalı T, Pearlson G, Yange H, Sponheim S, White T, Calhoun V (2010) A CCA + ICA based model for multi-task brain imaging data fusion and its application to schizophrenia. NeuroImage 51(1):123–134. PubMedPubMedCentralCrossRefGoogle Scholar
  60. Sui J, Pearlson G, Caprihan A, Adalı T, Kiehl K, Liu J, Yamamoto J, Calhoun VD (2011) Discriminating schizophrenia and bipolar disorder by fusing fMRI and DTI in a multimodal CCA + joint ICA model. NeuroImage 57(3):839–855. PubMedPubMedCentralCrossRefGoogle Scholar
  61. Sui J, He H, Yu Q, Chen J, Rogers J, Pearlson G, Mayer A, Bustillo J, Canive J, Calhoun VD (2013) Combination of resting state fMRI, DTI and sMRI data to discriminate schizophrenia by N-way MCCA+jICA. Front Hum Neurosci 7(235).
  62. Szabó Z, Póczos B, Lőrincz A (2012) Separation theorem for independent subspace analysis and its consequences. Pattern Recognit 45(4):1782–1791. CrossRefGoogle Scholar
  63. Ulloa A, Plis S, Erhardt E, Calhoun V (2015) Synthetic structural magnetic resonance image generator improves deep learning prediction of schizophrenia. In: Proc IEEE MLSP 2015, pp 1–6. Google Scholar
  64. Ulloa A, Plis SM, Calhoun VD (2018) Improving classification rate of schizophrenia using a multimodal multi-layer perceptron model with structural and functional MR. arXiv preprint URL
  65. Vergara VM, Ulloa A, Calhoun VD, Boutte D, Chen J, Liu J (2014) A three-way parallel ICA approach to analyze links among genetics, brain structure and brain function. NeuroImage 98:386–394. PubMedPubMedCentralCrossRefGoogle Scholar
  66. Wang P, Chen K, Yao L, Hu B, Wu X, Zhang J, Ye Q, Guo X (2016) Multimodal classification of mild cognitive impairment based on partial least squares. J Alzheimers Dis 54(1):359–371. PubMedCrossRefGoogle Scholar
  67. Wold H (1966) Nonlinear estimation by iterative least squares procedures. In: David F (ed) Research papers in statistics. Festschrift for J. Neyman. Wiley, New York, NY, pp 411–444Google Scholar
  68. Yeredor A (2000) Blind separation of gaussian sources via second-order statistics with asymptotically optimal weighting. IEEE Signal Process Lett 7(7):197–200. CrossRefGoogle Scholar
  69. Zhou YT, Chellappa R (1988) Computation of optical flow using a neural network. In: Proc IEEE ICNN 1988, vol 2, pp 71–78. Google Scholar

Copyright information

© Springer Nature Switzerland AG 2019

Authors and Affiliations

  1. 1.The Mind Research NetworkAlbuquerqueUSA

Personalised recommendations