Autism spectrum disorders (ASDs) are heterogeneous neurodevelopmental conditions. In fMRI studies, including most machine learning studies seeking to distinguish ASD from typical developing (TD) samples, cohorts differing in gender and symptom severity composition are often treated statistically as one ‘ASD group.’ Using resting-state functional connectivity (FC) data, we implemented random forest to build diagnostic classifiers in four ASD samples including a total of 656 participants (NASD = 306, NTD = 350, ages 6–18). Groups were manipulated to titrate heterogeneity of gender and symptom severity and partially overlapped. Each sample differed on inclusionary criteria: (1) all genders, unrestricted severity range; (2) only male participants, unrestricted severity; (3) all genders, higher severity only; and (4) only male participants, higher severity. Each set consisted of 200 participants per group (ASD, TD; matched on age and head motion): 160 for training and 40 for validation. FMRI time series from 237 regions of interest (ROIs) were Pearson correlated in a 237 × 237 FC matrix, and classifiers were built using random forest in training samples. Classification accuracies in validation samples were 62.5%, 65%, 70%, and 73.75%, respectively, for samples 1–4. Connectivity within cingulo-opercular task control (COTC) network, and between COTC ROIs and default mode and dorsal attention network contributed overall most informative features, but features differed across sets. Findings suggest that diagnostic classifiers vary depending on ASD sample composition. Specifically, greater homogeneity of samples regarding gender and symptom severity enhances classifier performance. However, given the true heterogeneity of ASDs, performance metrics alone may not adequately reflect classifier utility.
This is a preview of subscription content,to check access.
Access this article
Similar content being viewed by others
American Psychiatric Association (2013) Diagnostic and statistical manual of mental disorders: Dsm-5tm, 5th ed. American Psychiatric Publishing, Inc
Lord C, Elsabbagh M, Baird G, Veenstra-Vanderweele J (2018) Autism spectrum disorder. Lancet 392(10146):508–520
Lenroot RK, Yeung PK (2013) Heterogeneity within autism spectrum disorders: what have we learned from neuroimaging studies? Front Hum Neurosci 7:733
Hull JV, Jacokes ZJ, Torgerson CM, Irimia A, Van Horn JD (2017) Resting-state functional connectivity in autism spectrum disorders: a review. Front Psychiatry 7:205
Zhang S, Li X, Lv J, Jiang X, Guo L, Liu T (2016) Characterizing and differentiating task-based and resting state fmri signals via two-stage sparse representations. Brain Imaging Behav 10(1):21–32
Mastrovito D, Hanson C, Hanson SJ (2018) Differences in atypical resting-state effective connectivity distinguish autism from schizophrenia. Neuroimage Clin 18:367–376
Wang CH, Xiao ZY, Wu JH (2019) Functional connectivity-based classification of autism and control using svm-rfecv on rs-fmri data. Phys Med Eur J Med Phys 65:99–105
Sen B, Borle NC, Greiner R, Brown MRG (2018) A general prediction model for the detection of adhd and autism using structural and functional mri. PLoS ONE 13(4):e0194856
Chen H, Duan XJ, Liu F, Lu FM, Ma XJ, Zhang YX, Uddin LQ, Chen HF (2016) Multivariate classification of autism spectrum disorder using frequency-specific resting-state functional connectivity-a multi-center study. Prog Neuropsychopharmacol Biol Psychiatry 64:1–9
Feczko E, Balba NM, Miranda-Dominguez O, Cordova M, Karalunas SL, Irwin L, Demeter DV, Hill AP, Langhorst BH, Painter JG, Van Santen J, Fombonne EJ, Nigg JT, Fair DA (2018) Subtyping cognitive profiles in autism spectrum disorder using a functional random forest algorithm. Neuroimage 172:674–688
Jahedi A, Nasamran CA, Faires B, Fan J, Muller RA (2017) Distributed intrinsic functional connectivity patterns predict diagnostic status in large autism cohort. Brain Connect 7(8):515–525
Breiman L (2001) Random forests. Mach Learn 45(1):5–32
King JB, Prigge MBD, King CK, Morgan J, Weathersby F, Fox JC, Dean DC 3rd, Freeman A, Villaruz JAM, Kane KL, Bigler ED, Alexander AL, Lange N, Zielinski B, Lainhart JE, Anderson JS (2019) Generalizability and reproducibility of functional connectivity in autism. Mol Autism 10:27
Lombardo MV, Lai MC, Baron-Cohen S (2019) Big data approaches to decomposing heterogeneity across the autism spectrum. Mol Psychiatry 24(10):1435–1450
Alaerts K, Swinnen SP, Wenderoth N (2016) Sex differences in autism: a resting-state fmri investigation of functional brain connectivity in males and females. Soc Cogn Affect Neurosci 11(6):1002–1016
Floris DL, Lai MC, Nath T, Milham MP, Di Martino A (2018) Network-specific sex differentiation of intrinsic brain function in males with autism. Mol Autism 9:17
Olson LA, Mash LE, Linke A, Fong CH, Mueller RA, Fishman I. Sex-related patterns of intrinsic functional connectivity in children and adolescents with autism spectrum disorders. Autism (in press)
Smith REW, Avery JA, Wallace GL, Kenworthy L, Gotts SJ, Martin A (2019) Sex differences in resting-state functional connectivity of the cerebellum in autism spectrum disorder. Front Hum Neurosci 13:104
Di Martino A, O’Connor D, Chen B, Alaerts K, Anderson JS, Assaf M, Balsters JH, Baxter L, Beggiato A, Bernaerts S, Blanken LM, Bookheimer SY, Braden BB, Byrge L, Castellanos FX, Dapretto M, Delorme R, Fair DA, Fishman I, Fitzgerald J, Gallagher L, Keehn RJ, Kennedy DP, Lainhart JE, Luna B, Mostofsky SH, Muller RA, Nebel MB, Nigg JT, O’Hearn K, Solomon M, Toro R, Vaidya CJ, Wenderoth N, White T, Craddock RC, Lord C, Leventhal B, Milham MP (2017) Enhancing studies of the connectome in autism using the autism brain imaging data exchange II. Sci Data 4:170010
Di Martino A, Yan CG, Li Q, Denio E, Castellanos FX, Alaerts K, Anderson JS, Assaf M, Bookheimer SY, Dapretto M, Deen B, Delmonte S, Dinstein I, Ertl-Wagner B, Fair DA, Gallagher L, Kennedy DP, Keown CL, Keysers C, Lainhart JE, Lord C, Luna B, Menon V, Minshew NJ, Monk CS, Mueller S, Muller RA, Nebel MB, Nigg JT, O’Hearn K, Pelphrey KA, Peltier SJ, Rudie JD, Sunaert S, Thioux M, Tyszka JM, Uddin LQ, Verhoeven JS, Wenderoth N, Wiggins JL, Mostofsky SH, Milham MP (2014) The autism brain imaging data exchange: towards a large-scale evaluation of the intrinsic brain architecture in autism. Mol Psychiatry 19(6):659–667
Lord C, Rutter M, Goode S, Heemsbergen J, Jordan H, Mawhood L, Schopler E (1989) Autism diagnostic observation schedule: a standardized observation of communicative and social behavior. J Autism Dev Disord 19(2):185–212
Lord C, Rutter M, DiLavore PC, Risi S, Gotham K, Bishop SL (2012) Autism diagnostic observation schedule, 2nd edn. Western Psychological Services, Torrance
Lord C, Rutter M, Le Couteur A (1994) Autism diagnostic interview-revised: a revised version of a diagnostic interview for caregivers of individuals with possible pervasive developmental disorders. J Autism Dev Disord 24(5):659–685
Nair S, Jao Keehn RJ, Berkebile MM, Maximo JO, Witkowska N, Muller RA (2018) Local resting state functional connectivity in autism: site and cohort variability and the effect of eye status. Brain Imaging Behav 12(1):168–179
Power JD, Mitra A, Laumann TO, Snyder AZ, Schlaggar BL, Petersen SE (2014) Methods to detect, characterize, and remove motion artifact in resting state fmri. Neuroimage 84:320–341
Loomes R, Hull L, Mandy WPL (2017) What is the male-to-female ratio in autism spectrum disorder? A systematic review and meta-analysis. J Am Acad Child Adolesc Psychiatry 56(6):466–474
Jahedi A, Hills T, Olson M, Müller R-A, Fan J. Iterative multivariate matching package for samples with missing data: The iterMatch Package for R (under review)
Cox RW (1996) Afni: software for analysis and visualization of functional magnetic resonance neuroimages. Comput Biomed Res 29(3):162–173
Smith SM, Jenkinson M, Woolrich MW, Beckmann CF, Behrens TE, Johansen-Berg H, Bannister PR, De Luca M, Drobnjak I, Flitney DE, Niazy RK, Saunders J, Vickers J, Zhang Y, De Stefano N, Brady JM, Matthews PM (2004) Advances in functional and structural mr image analysis and implementation as fsl. Neuroimage 23(Suppl 1):S208–S219
Keown CL, Datko MC, Chen CP, Maximo JO, Jahedi A, Muller RA (2017) Network organization is globally atypical in autism: a graph theory study of intrinsic functional connectivity. Biol Psychiatry Cognit Neurosci Neuroimaging 2(1):66–75
Hallquist MN, Hwang K, Luna B (2013) The nuisance of nuisance regression: spectral misspecification in a common approach to resting-state fmri preprocessing reintroduces noise and obscures functional connectivity. Neuroimage 82:208–225
Gordon EM, Laumann TO, Adeyemo B, Huckins JF, Kelley WM, Petersen SE (2016) Generation and evaluation of a cortical area parcellation from resting-state correlations. Cereb Cortex 26(1):288–303
Desikan RS, Segonne F, Fischl B, Quinn BT, Dickerson BC, Blacker D, Buckner RL, Dale AM, Maguire RP, Hyman BT, Albert MS, Killiany RJ (2006) An automated labeling system for subdividing the human cerebral cortex on mri scans into gyral based regions of interest. Neuroimage 31(3):968–980
Diedrichsen J, Balsters JH, Flavell J, Cussans E, Ramnani N (2009) A probabilistic mr atlas of the human cerebellum. Neuroimage 46(1):39–46
Hothorn T, Hornik K, Zeileis A (2006) Unbiased recursive partitioning: a conditional inference framework. J Comput Gr Stat 15(3):651–674
R Core Team (2013) R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna
Liaw A, Wiener M (2002) Classification and regression by random forest. R News 2(3):18–22
Strobl C, Boulesteix AL, Kneib T, Augustin T, Zeileis A (2008) Conditional variable importance for random forests. BMC Bioinform 9:307
He Y, Byrge L, Kennedy DP (2020) Nonreplication of functional connectivity differences in autism spectrum disorder across multiple sites and denoising strategies. Hum Brain Mapp 41(5):1334–1350
Abraham A, Milham MP, Di Martino A, Craddock RC, Samaras D, Thirion B, Varoquaux G (2017) Deriving reproducible biomarkers from multi-site resting-state data: an autism-based example. Neuroimage 147:736–745
Anderson JS, Nielsen JA, Froehlich AL, DuBray MB, Druzgal TJ, Cariello AN, Cooperrider JR, Zielinski BA, Ravichandran C, Fletcher PT, Alexander AL, Bigler ED, Lange N, Lainhart JE (2011) Functional connectivity magnetic resonance imaging classification of autism. Brain 134(Pt 12):3742–3754
Kassraian-Fard P, Matthis C, Balsters JH, Maathuis MH, Wenderoth N (2016) Promises, pitfalls, and basic guidelines for applying machine learning classifiers to psychiatric imaging data, with autism as an example. Front Psychiatry 7:177
Yahata N, Morimoto J, Hashimoto R, Lisi G, Shibata K, Kawakubo Y, Kuwabara H, Kuroda M, Yamada T, Megumi F, Imamizu H, Nanez JE, Takahashi H, Okamoto Y, Kasai K, Kato N, Sasaki Y, Watanabe T, Kawato M (2016) A small number of abnormal brain connections predicts adult autism spectrum disorder. Nat Commun 7:1–2
Plitt M, Barnes KA, Martin A (2015) Functional connectivity classification of autism identifies highly predictive brain features but falls short of biomarker standards. Neuroimage Clin 7:359–366
Lombardo MV, Eyler L, Moore A, Datko M, Carter Barnes C, Cha D, Courchesne E, Pierce K (2019) Default mode-visual network hypoconnectivity in an autism subtype with pronounced social visual engagement difficulties. Elife 8:e47427
Lynn AC, Padmanabhan A, Simmonds D, Foran W, Hallquist MN, Luna B, O’Hearn K (2018) Functional connectivity differences in autism during face and car recognition: underconnectivity and atypical age-related changes. Dev Sci 21(1):e12508
Samson F, Mottron L, Soulieres I, Zeffiro TA (2012) Enhanced visual functioning in autism: an ale meta-analysis. Hum Brain Mapp 33(7):1553–1581
Reiter, M. A., Mash, L. E., Linke, A. C., Fong, C. H., Fishman, I., & Muller, R. A. (2018). Distinct Patterns of Atypical Functional Connectivity in Lower-Functioning Autism. Biol Psychiatry Cogn Neurosci Neuroimaging, 4(3), 251-259.
Farrant K, Uddin LQ (2016) Atypical developmental of dorsal and ventral attention networks in autism. Dev Sci 19(4):550–563
Nomi JS, Uddin LQ (2015) Developmental changes in large-scale network connectivity in autism. Neuroimage Clin 7:732–741
This study was supported by the National Institutes of Health R01 MH081023 (RAM) and R01 MH101173 (RAM). Authors MAR and AJ were supported by San Diego State University Graduate Fellowships. MAR was also supported by the Autism Speaks Sir Dennis Weatherstone Fellowship program (Grant No. 10609). ARJF was supported by the Science Engineering and Research Board (SERB)/Indo-Us Science and Technology Forum (IUSSTF), Grant 119. We would like to acknowledge here the participants and families who enrolled in the current study, without whom the current project would not have been possible.
Conflicts of interest
The authors have no known conflicts of interest.
All data collection and sharing were conducted under the approval of the appropriate Institutional Review Boards.
Consent to participate
All participants provided written consent to participate and share their de-identified data for the purposes of research.
Consent for publication
All authors were involved in the preparation of this manuscript and consent to its publication.
Availability of data and material
All of the data included in the current study is publically available through the Autism Brain Imaging Data Exchange (ABIDE) initiative online.
A combination of publically available neuroimaging processing software, statistical analysis software, and custom code generated by the research team was used in the preparation of this manuscript.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
Below is the link to the electronic supplementary material.
About this article
Cite this article
Reiter, M.A., Jahedi, A., Fredo, A.R.J. et al. Performance of machine learning classification models of autism using resting-state fMRI is contingent on sample heterogeneity. Neural Comput & Applic 33, 3299–3310 (2021). https://doi.org/10.1007/s00521-020-05193-y