Abstract
Resting-state functional connectivity analysis has become a widely used method for the investigation of human brain connectivity and pathology. The measurement of neuronal activity by functional MRI, however, is impeded by various nuisance signals that reduce the stability of functional connectivity. Several methods exist to address this predicament, but little consensus has yet been reached on the most appropriate approach. Given the crucial importance of reliability for the development of clinical applications, we here investigated the effect of various confound removal approaches on the test–retest reliability of functional-connectivity estimates in two previously defined functional brain networks. Our results showed that gray matter masking improved the reliability of connectivity estimates, whereas denoising based on principal components analysis reduced it. We additionally observed that refraining from using any correction for global signals provided the best test–retest reliability, but failed to reproduce anti-correlations between what have been previously described as antagonistic networks. This suggests that improved reliability can come at the expense of potentially poorer biological validity. Consistent with this, we observed that reliability was proportional to the retained variance, which presumably included structured noise, such as reliable nuisance signals (for instance, noise induced by cardiac processes). We conclude that compromises are necessary between maximizing test–retest reliability and removing variance that may be attributable to non-neuronal sources.
Similar content being viewed by others
References
Amft M, Bzdok D, Laird AR et al (2015) Definition and characterization of an extended social-affective default network. Brain Struct Funct 220:1031–1049. doi:10.1007/s00429-013-0698-0
Ashburner J, Friston KJ (2005) Unified segmentation. Neuroimage 26:839–851. doi:10.1016/j.neuroimage.2005.02.018
Barch DM, Burgess GC, Harms MP et al (2013) Function in the human connectome: task-fMRI and individual differences in behavior. Neuroimage 80:169–189. doi:10.1016/j.neuroimage.2013.05.033
Behzadi Y, Restom K, Liau J, Liu TT (2007) A component based noise correction method (CompCor) for BOLD and perfusion based fMRI. Neuroimage 37:90–101. doi:10.1016/j.neuroimage.2007.04.042
Birn RM, Diamond JB, Smith MABP (2006) Separating respiratory-variation-related fluctuations from neuronal-activity-related fluctuations in fMRI. Neuroimage 31:1536–1548
Birn RM, Molloy EK, Patriat R et al (2013) The effect of scan length on the reliability of resting-state fMRI connectivity estimates. Neuroimage 83:550–558. doi:10.1016/j.neuroimage.2013.05.099
Birn RM, Cornejo MD, Molloy EK et al (2014) The influence of physiological noise correction on test–retest reliability of resting-state functional connectivity. Brain Connect 4:511–522. doi:10.1089/brain.2014.0284
Biswal B, Yetkin FZ, Haughton VM, Hyde JS (1995) Functional connectivity in the motor cortex of resting human brain using echo-planar MRI. Magn Reson Med 34:537–541. doi:10.1002/mrm.1910340409
Biswal BB, Van Kylen J, Hyde JS (1997) Simultaneous assessment of flow and BOLD signals in resting-state functional connectivity maps. NMR Biomed 10(4–5):165–170
Buckner RL (2010) Human functional connectivity: new tools, unresolved questions. Proc Natl Acad Sci 107(24):10769–10770
Buckner RL, Bandettini PA, O’Craven KM et al (1996) Detection of cortical activation during averaged single trials of a cognitive task using functional magnetic resonance imaging. Proc Natl Acad Sci USA 93:14878–14883. doi:10.1073/pnas.93.25.14878
Button KS, Ioannidis JPA, Mokrysz C et al (2013a) Power failure: why small sample size undermines the reliability of neuroscience. Nat Rev Neurosci 14:365–376. doi:10.1038/nrn3475
Button KS, Ioannidis JPA, Mokrysz C et al (2013b) Empirical evidence for low reproducibility indicates low pre-study odds. Nat Rev Neurosci 14:877. doi:10.1038/nrn3475-c6
Bzdok D, Laird AR, Zilles K et al (2013) An investigation of the structural, connectional, and functional subspecialization in the human amygdala. Hum Brain Mapp 34:3247–3266. doi:10.1002/hbm.22138
Chai XJ, Castañón AN, Öngür D, Whitfield-Gabrieli S (2012) Anticorrelations in resting state networks without global signal regression. Neuroimage 59:1420–1428. doi:10.1016/j.neuroimage.2011.08.048
Chang C, Cunningham JP, Glover GH (2009) Influence of heart rate on the BOLD signal: the cardiac response function. Neuroimage 44:857–869. doi:10.1016/j.neuroimage.2008.09.029
Chen G, Chen G, Xie C et al (2012) A method to determine the necessity for global signal regression in resting-state fMRI studies. Magn Reson Med 68:1828–1835. doi:10.1002/mrm.24201
Chen B, Xu T, Zhou C et al (2015) Individual variability and test–retest reliability revealed by ten repeated resting-state brain scans over one month. PLoS One 10:e0144963. doi:10.1371/journal.pone.0144963
Clos M, Amunts K, Laird AR et al (2013) Tackling the multifunctional nature of Broca’s region meta-analytically: co-activation-based parcellation of area 44. Neuroimage 83:174–188. doi:10.1016/j.neuroimage.2013.06.041
Cole DM, Smith SM, Beckmann CF (2010) Advances and pitfalls in the analysis and interpretation of resting-state FMRI data. Front Syst Neurosci 4:8. doi:10.3389/fnsys.2010.00008
Cordes D, Haughton VM, Arfanakis K et al (2001) Frequencies contributing to functional connectivity in the cerebral cortex in HResting-state data. Am J Neuroradiol 22:1326–1333
Eickhoff SB, Stephan KE, Mohlberg H et al (2005) A new SPM toolbox for combining probabilistic cytoarchitectonic maps and functional imaging data. Neuroimage 25:1325–1335. doi:10.1016/j.neuroimage.2004.12.034
Eickhoff SB, Laird AR, Grefkes C et al (2009) Coordinate-based activation likelihood estimation meta-analysis of neuroimaging data: a random-effects approach based on empirical estimates of spatial uncertainty. Hum Brain Mapp 30:2907–2926. doi:10.1002/hbm.20718
Eickhoff SB, Bzdok D, Laird AR et al (2011) Co-activation patterns distinguish cortical modules, their connectivity and functional differentiation. Neuroimage 57:938–949. doi:10.1016/j.neuroimage.2011.05.021
Eickhoff SB, Bzdok D, Laird AR et al (2012) Activation likelihood estimation meta-analysis revisited. Neuroimage 59:2349–2361. doi:10.1016/j.neuroimage.2011.09.017
Eickhoff SB, Thirion B, Varoquaux G, Bzdok D (2015) Connectivity-based parcellation: critique and implications. Hum Brain Mapp 36:4771–4792
Fox MD, Raichle ME (2007) Spontaneous fluctuations in brain activity observed with functional magnetic resonance imaging. Nat Rev Neurosci 8:700–711. doi:10.1038/nrn2201
Fox MD, Snyder AZ, Vincent JL et al (2005) The human brain is intrinsically organized into dynamic, anticorrelated functional networks. Proc Natl Acad Sci USA 102:9673–9678. doi:10.1073/pnas.0504136102
Fox MD, Zhang D, Snyder AZ, Raichle ME (2009) The global signal and observed anticorrelated resting state brain networks. J Neurophysiol 101:3270–3283. doi:10.1152/jn.90777.2008
Fox MD, Zhang D, Snyder AZ, Raichle ME (2013) The global signal and observed anticorrelated resting state brain networks 101:3270–3283. doi:10.1152/jn.90777.2008
Fox PT, Lancaster JL, Laird AR, Eickhoff SB (2014) Meta-analysis in human neuroimaging: computational modeling of large-scale databases. Annu Rev Neurosci 37:409–434. doi:10.1146/annurev-neuro-062012-170320
Frässle S, Paulus FM, Krach S, Jansen A (2016) Test–retest reliability of effective connectivity in the face perception network. Hum Brain Mapp 744:730–744. doi:10.1002/hbm.23061
Friston KJ (2011) Functional and effective connectivity: a review. Brain Connect 1:13–36. doi:10.1089/brain.2011.0008
Friston KJ, Williams S, Howard R et al (1996) Movement-related effects in fMRI time-series. Magn Reson Med 35:346–355. doi:10.1002/mrm.1910350312
Friston KJ, Rotshtein P, Geng JJ et al (2006) A critique of functional localisers. Neuroimage 30:1077–1087. doi:10.1016/j.neuroimage.2005.08.012
Gorgolewski KJ, Storkey AJ, Bastin ME et al (2013) Single subject fMRI test–retest reliability metrics and confounding factors. Neuroimage 69:231–243. doi:10.1016/j.neuroimage.2012.10.085
Greicius MD, Menon V (2004) Default-mode activity during a passive sensory task: uncoupled from deactivation but impacting activation. J Cogn Neurosci 16:1484–1492. doi:10.1162/0898929042568532
Greicius MD, Krasnow B, Reiss AL, Menon V (2003) Functional connectivity in the resting brain: a network analysis of the default mode hypothesis. Proc Natl Acad Sci USA 100:253–258. doi:10.1073/pnas.0135058100
Griffanti L, Salimi-Khorshidi G, Beckmann CF et al (2014) ICA-based artefact removal and accelerated fMRI acquisition for improved resting state network imaging. Neuroimage 95:232–247. doi:10.1016/j.neuroimage.2014.03.034
Guo WB, Liu F, Xue ZM et al (2011) Abnormal neural activities in first-episode, treatment-naïve, short-illness-duration, and treatment-response patients with major depressive disorder: a resting-state fMRI study. J Affect Disord 135:326–331. doi:10.1016/j.jad.2011.06.048
He B, Wang Y, Wu D (1999) Estimating cortical potentials from scalp EEG’s in a realistically shaped inhomogeneous head model by means of the boundary element method. IEEE Trans Biomed Eng 46:1264–1268. doi:10.1109/10.790505
Hinke RM, Hu X, Stillman AE et al (1993) Functional magnetic resonance imaging of Broca’s area during internal speech. NeuroReport 4:675–678
Hoptman MJ, Zuo X-N, D’Angelo D et al (2012) Decreased interhemispheric coordination in schizophrenia: a resting state fMRI study. Schizophr Res 141:1–7. doi:10.1016/j.schres.2012.07.027
Huettel SA, Song AW, McCarthy G (2004) Functional magnetic resonance imaging. Sinauer Associates Publishers, Sunderland
Jo HJ, Saad ZS, Simmons WK et al (2010) Mapping sources of correlation in resting state FMRI, with artifact detection and removal. Neuroimage 52:571–582. doi:10.1016/j.neuroimage.2010.04.246
Jo HJ, Gotts SJ, Reynolds RC et al (2013) Effective preprocessing procedures virtually eliminate distance-dependent motion artifacts in resting state FMRI. J Appl Math. doi:10.1155/2013/935154
Kellermann TS, Caspers S, Fox PT et al (2013) Task- and resting-state functional connectivity of brain regions related to affection and susceptible to concurrent cognitive demand. Neuroimage 72:69–82. doi:10.1016/j.neuroimage.2013.01.046
Kim S-G, Ogawa S (2012) Biophysical and physiological origins of blood oxygenation level-dependent fMRI signals. J Cereb Blood Flow Metab 32:1188–1206. doi:10.1038/jcbfm.2012.23
Kwong KK, Belliveau JW, Chesler DA et al (1992) Dynamic magnetic resonance imaging of human brain activity during primary sensory stimulation. Proc Natl Acad Sci USA 89:5675–5679. doi:10.1073/pnas.89.12.5675
Lai Y, Zhang X, van Drongelen W et al (2011) Noninvasive cortical imaging of epileptiform activities from interictal spikes in pediatric patients. Neuroimage 54:244–252. doi:10.1016/j.neuroimage.2010.07.026
Li Z, Kadivar A, Pluta J et al (2012) Test–retest stability analysis of resting brain activity revealed by blood oxygen level-dependent functional MRI. J Magn Reson Imaging 36:344–354. doi:10.1002/jmri.23670
Lu Y, Worrell GA, Zhang HC et al (2014) Noninvasive imaging of the high frequency brain activity in focal epilepsy patients. IEEE Trans Biomed Eng 61:1660–1667. doi:10.1109/TBME.2013.2297332
Macey PM, Macey KE, Kumar R, Harper RM (2004) A method for removal of global effects from fMRI time series. Neuroimage 22:360–366. doi:10.1016/j.neuroimage.2003.12.042
Müller VI, Cieslik EC, Laird AR et al (2013) Dysregulated left inferior parietal activity in schizophrenia and depression: functional connectivity and characterization. Front Hum Neurosci 7:268. doi:10.3389/fnhum.2013.00268
Müller VI, Langner R, Cieslik EC et al (2014) Interindividual differences in cognitive flexibility: influence of gray matter volume, functional connectivity and trait impulsivity. Brain Struct Funct 220:2401–2414. doi:10.1007/s00429-014-0797-6
Murphy K, Birn RM, Handwerker DA et al (2009) The impact of global signal regression on resting state correlations: are anti-correlated networks introduced? Neuroimage 44:893–905. doi:10.1016/j.neuroimage.2008.09.036
Niazy RK, Xie J, Miller K et al (2011) Spectral characteristics of resting state networks, 1st edn. Elsevier B.V, Amsterdam
Patriat R, Molloy EK, Meier TB et al (2013) The effect of resting condition on resting-state fMRI reliability and consistency: a comparison between resting with eyes open, closed, and fixated. Neuroimage 78:463–473. doi:10.1016/j.neuroimage.2013.04.013
Patriat R, Molloy EK, Birn R (2015) Using edge voxel information to improve motion regression for rs-fMRI connectivity studies. Brain Connect 5:582–595. doi:10.1089/brain.2014.0321
Pisauro MA, Benucci A, Carandini M (2016) Local and global contributions to hemodynamic activity in mouse cortex. J Neurophysiol 115:2931–2936. doi:10.1152/jn.00125.2016
Power JD, Barnes KA, Snyder AZ, Schlaggar BL, Petersen SE (2012) Spurious but systematic correlations in functional connectivity MRI networks arise from subject motion. Neuroimage 59:2142–2154. doi:10.1016/j.neuroimage.2011.10.018.Spurious
Power JD, Mitra A, Laumann TO et al (2014) Methods to detect, characterize, and remove motion artifact in resting state fMRI. Neuroimage 84:320–341. doi:10.1016/j.neuroimage.2013.08.048
Power JD, Schlaggar BL, Petersen SE (2015) Recent progress and outstanding issues in motion correction in resting state fMRI. Neuroimage 105:536–551. doi:10.1016/j.neuroimage.2014.10.044
Pruim RHR, Mennes M, Buitelaar JK, Beckmann CF (2015a) Evaluation of ICA-AROMA and alternative strategies for motion artifact removal in resting state fMRI. Neuroimage 112:278–287. doi:10.1016/j.neuroimage.2015.02.063
Pruim RHR, Mennes M, van Rooij D et al (2015b) ICA-AROMA: a robust ICA-based strategy for removing motion artifacts from fMRI data. Neuroimage 112:267–277. doi:10.1016/j.neuroimage.2015.02.064
Raemaekers M, Du Plessis S, Ramsey NF et al (2012) Test–retest variability underlying fMRI measurements. Neuroimage 60:717–727. doi:10.1016/j.neuroimage.2011.11.061
Reid AT, Evans AC (2013) Structural networks in Alzheimer’s disease. Eur Neuropsychopharmacol 23:63–77. doi:10.1016/j.euroneuro.2012.11.010
Reid AT, Bzdok D, Langner R et al (2016) Multimodal connectivity mapping of the human left anterior and posterior lateral prefrontal cortex. Brain Struct Funct 221:2589–2605
Rottschy C, Langner R, Dogan I et al (2012) Modelling neural correlates of working memory: a coordinate-based meta-analysis. Neuroimage 60:830–846. doi:10.1016/j.neuroimage.2011.11.050
Saad ZS, Gotts SJ, Murphy K et al (2012) Trouble at rest: how correlation patterns and group differences become distorted after global signal regression. Brain Connect 2:25–32. doi:10.1089/brain.2012.0080
Salimi-Khorshidi G, Douaud G, Beckmann CF et al (2014) Automatic denoising of functional MRI data: combining independent component analysis and hierarchical fusion of classifiers. Neuroimage 90:449–468. doi:10.1016/j.neuroimage.2013.11.046
Satterthwaite TD, Elliott MA, Gerraty RT, Ruparel K, Loughead J, Calkins ME, Eickhoff SB, Hakonarson H, Gut RC, Gur REWD (2013) An improved framework for confound regression and filtering for control of motion artifact in the preprocessing of resting-state functional connectivity data. Neuroimage 64:240–256
Schilbach L (2016) Differential patterns of dysconnectivity in mirror neuron and mentalizing networks in schizophrenia. Schizophr Bull. doi:10.1093/schbul/sbw015
Schilbach L, Bzdok D, Timmermans B et al (2012) Introspective minds: using ALE meta-analyses to study commonalities in the neural correlates of emotional processing, social and unconstrained cognition. PLoS One 7:e30920. doi:10.1371/journal.pone.0030920
Schilbach L, Müller VI, Hoffstaedter F et al (2014) Meta-analytically informed network analysis of resting state fMRI reveals hyperconnectivity in an introspective socio-affective network in depression. PLoS One 9:e94973. doi:10.1371/journal.pone.0094973
Schölvinck ML, Leopold DA, Brookes MJ, Khader PH (2013) The contribution of electrophysiology to functional connectivity mapping. Neuroimage 80:297–306. doi:10.1016/j.neuroimage.2013.04.010
Shehzad Z, Kelly AMC, Reiss PT et al (2009) The resting brain: unconstrained yet reliable. Cereb Cortex 19:2209–2229. doi:10.1093/cercor/bhn256
Shen X, Papademetris X, Constable RT (2010) Graph-theory based parcellation of functional subunits in the brain from resting-state fMRI data. Neuroimage 50:1027–1035. doi:10.1016/j.neuroimage.2009.12.119
Shirer WR, Jiang H, Price CM et al (2015) Optimization of rs-fMRI pre-processing for enhanced signal-noise separation, test–retest reliability, and group discrimination. Neuroimage 117:67–79. doi:10.1016/j.neuroimage.2015.05.015
Smith DV, Utevsky AV, Bland AR et al (2014) Characterizing individual differences in functional connectivity using dual-regression and seed-based approaches. Neuroimage 95:1–12. doi:10.1016/j.neuroimage.2014.03.042
Soltysik DA, Thomasson D, Rajan S, Biassou N (2015) Improving the use of principal component analysis to reduce physiological noise and motion artifacts to increase the sensitivity of task-based fMRI. J Neurosci Methods 241:18–29. doi:10.1016/j.jneumeth.2014.11.015
Song X, Lawrence PP, Chen N-K (2016) Data-driven and predefined ROI-based quantification of long-term resting-state fMRI reproducibility. Brain Connect XX:1–16. doi:10.1089/brain.2015.0349
Thomason ME, Dennis EL, Joshi AA et al (2011) Resting-state fMRI can reliably map neural networks in children. Neuroimage 55:165–175. doi:10.1016/j.neuroimage.2010.11.080
Tsvetanov KA, Henson RNA, Tyler LK et al (2015) The effect of ageing on fMRI: correction for the confounding effects of vascular reactivity evaluated by joint fMRI and MEG in 335 adults. Hum Brain Mapp 36:2248–2269. doi:10.1002/hbm.22768
Van Dijk KR, Sabuncu MR, Buckner RL (2012) The influence of head motion on intrinsic functional connectivity MRI. Neuroimage 59:431–438. doi:10.1016/j.neuroimage.2011.07.044.The
Wang J-H, Zuo X-N, Gohel S et al (2011) Graph theoretical analysis of functional brain networks: test–retest evaluation on short- and long-term resting-state functional MRI data. PLoS One 6:e21976. doi:10.1371/journal.pone.0021976
Weissenbacher A, Kasess C, Gerstl F et al (2009) Correlations and anticorrelations in resting-state functional connectivity MRI: a quantitative comparison of preprocessing strategies. Neuroimage 47:1408–1416. doi:10.1016/j.neuroimage.2009.05.005
Windischberger C, Langenberger H, Sycha T et al (2002) On the origin of respiratory artifacts in BOLD-EPI of the human brain. Magn Reson Imaging 20:575–582. doi:10.1016/S0730-725X(02)00563-5
Wong C-K, Zotev V, Misaki M et al (2016) Automatic EEG-assisted retrospective motion correction for fMRI (aE-REMCOR). Neuroimage 129:133–147. doi:10.1016/j.neuroimage.2016.01.042
Yan CG, Cheung B, Kelly C et al (2013a) A comprehensive assessment of regional variation in the impact of head micromovements on functional connectomics. Neuroimage 76:183–201. doi:10.1016/j.neuroimage.2013.03.004
Yan CG, Craddock RC, Zuo XN et al (2013b) Standardizing the intrinsic brain: towards robust measurement of inter-individual variation in 1000 functional connectomes. Neuroimage 80:246–262. doi:10.1016/j.neuroimage.2013.04.081
Zang Y, Jiang T, Lu Y et al (2004) Regional homogeneity approach to fMRI data analysis. Neuroimage 22:394–400. doi:10.1016/j.neuroimage.2003.12.030
Zhang D, Raichle ME (2010) Disease and the brains dark energy. Nat Rev Neurol 6:15–28
Zhong S, He Y, Gong G (2015) Convergence and divergence across construction methods for human brain white matter networks: an assessment based on individual differences. Hum Brain Mapp 36:1995–2013. doi:10.1002/hbm.22751
Zou QH, Zhu CZ, Yang Y et al (2008) An improved approach to detection of amplitude of low-frequency fluctuation (ALFF) for resting-state fMRI: fractional ALFF. J Neurosci Methods 172:137–141. doi:10.1016/j.jneumeth.2008.04.012
Zuo XN, Kelly C, Adelstein JS et al (2010) Reliable intrinsic connectivity networks: test–retest evaluation using ICA and dual regression approach. Neuroimage 49:2163–2177. doi:10.1016/j.neuroimage.2009.10.080
Acknowledgments
This study was supported by the Deutsche Forschungsgemeinschaft (DFG, EI 816/4-1, LA 3071/3-1; EI 816/6-1), the National Institute of Mental Health (R01-MH074457), the Helmholtz Portfolio Theme “Supercomputing and Modeling for the Human Brain” and the European Union Seventh Framework Program (FP7/2007–2013) under Grant Agreement No. 604102 (Human Brain Project).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Ethical approval
The original study protocol of the data used here has been approved by the local ethics committees of the university hospital Aachen and informed consent was obtained by all the participants prior to the examination. The current data were analyzed anonymously.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Varikuti, D.P., Hoffstaedter, F., Genon, S. et al. Resting-state test–retest reliability of a priori defined canonical networks over different preprocessing steps. Brain Struct Funct 222, 1447–1468 (2017). https://doi.org/10.1007/s00429-016-1286-x
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00429-016-1286-x