Skip to main content
Log in

Performance of machine learning classification models of autism using resting-state fMRI is contingent on sample heterogeneity

Neural Computing and Applications Aims and scope Submit manuscript

Cite this article

Abstract

Autism spectrum disorders (ASDs) are heterogeneous neurodevelopmental conditions. In fMRI studies, including most machine learning studies seeking to distinguish ASD from typical developing (TD) samples, cohorts differing in gender and symptom severity composition are often treated statistically as one ‘ASD group.’ Using resting-state functional connectivity (FC) data, we implemented random forest to build diagnostic classifiers in four ASD samples including a total of 656 participants (NASD = 306, NTD = 350, ages 6–18). Groups were manipulated to titrate heterogeneity of gender and symptom severity and partially overlapped. Each sample differed on inclusionary criteria: (1) all genders, unrestricted severity range; (2) only male participants, unrestricted severity; (3) all genders, higher severity only; and (4) only male participants, higher severity. Each set consisted of 200 participants per group (ASD, TD; matched on age and head motion): 160 for training and 40 for validation. FMRI time series from 237 regions of interest (ROIs) were Pearson correlated in a 237 × 237 FC matrix, and classifiers were built using random forest in training samples. Classification accuracies in validation samples were 62.5%, 65%, 70%, and 73.75%, respectively, for samples 1–4. Connectivity within cingulo-opercular task control (COTC) network, and between COTC ROIs and default mode and dorsal attention network contributed overall most informative features, but features differed across sets. Findings suggest that diagnostic classifiers vary depending on ASD sample composition. Specifically, greater homogeneity of samples regarding gender and symptom severity enhances classifier performance. However, given the true heterogeneity of ASDs, performance metrics alone may not adequately reflect classifier utility.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  1. American Psychiatric Association (2013) Diagnostic and statistical manual of mental disorders: Dsm-5tm, 5th ed. American Psychiatric Publishing, Inc

  2. Lord C, Elsabbagh M, Baird G, Veenstra-Vanderweele J (2018) Autism spectrum disorder. Lancet 392(10146):508–520

    Google Scholar 

  3. Lenroot RK, Yeung PK (2013) Heterogeneity within autism spectrum disorders: what have we learned from neuroimaging studies? Front Hum Neurosci 7:733

    Google Scholar 

  4. Hull JV, Jacokes ZJ, Torgerson CM, Irimia A, Van Horn JD (2017) Resting-state functional connectivity in autism spectrum disorders: a review. Front Psychiatry 7:205

    Google Scholar 

  5. Zhang S, Li X, Lv J, Jiang X, Guo L, Liu T (2016) Characterizing and differentiating task-based and resting state fmri signals via two-stage sparse representations. Brain Imaging Behav 10(1):21–32

    Google Scholar 

  6. Mastrovito D, Hanson C, Hanson SJ (2018) Differences in atypical resting-state effective connectivity distinguish autism from schizophrenia. Neuroimage Clin 18:367–376

    Google Scholar 

  7. Wang CH, Xiao ZY, Wu JH (2019) Functional connectivity-based classification of autism and control using svm-rfecv on rs-fmri data. Phys Med Eur J Med Phys 65:99–105

    Google Scholar 

  8. Sen B, Borle NC, Greiner R, Brown MRG (2018) A general prediction model for the detection of adhd and autism using structural and functional mri. PLoS ONE 13(4):e0194856

    Google Scholar 

  9. Chen H, Duan XJ, Liu F, Lu FM, Ma XJ, Zhang YX, Uddin LQ, Chen HF (2016) Multivariate classification of autism spectrum disorder using frequency-specific resting-state functional connectivity-a multi-center study. Prog Neuropsychopharmacol Biol Psychiatry 64:1–9

    Google Scholar 

  10. Feczko E, Balba NM, Miranda-Dominguez O, Cordova M, Karalunas SL, Irwin L, Demeter DV, Hill AP, Langhorst BH, Painter JG, Van Santen J, Fombonne EJ, Nigg JT, Fair DA (2018) Subtyping cognitive profiles in autism spectrum disorder using a functional random forest algorithm. Neuroimage 172:674–688

    Google Scholar 

  11. Jahedi A, Nasamran CA, Faires B, Fan J, Muller RA (2017) Distributed intrinsic functional connectivity patterns predict diagnostic status in large autism cohort. Brain Connect 7(8):515–525

    Google Scholar 

  12. Breiman L (2001) Random forests. Mach Learn 45(1):5–32

    MATH  Google Scholar 

  13. King JB, Prigge MBD, King CK, Morgan J, Weathersby F, Fox JC, Dean DC 3rd, Freeman A, Villaruz JAM, Kane KL, Bigler ED, Alexander AL, Lange N, Zielinski B, Lainhart JE, Anderson JS (2019) Generalizability and reproducibility of functional connectivity in autism. Mol Autism 10:27

    Google Scholar 

  14. Lombardo MV, Lai MC, Baron-Cohen S (2019) Big data approaches to decomposing heterogeneity across the autism spectrum. Mol Psychiatry 24(10):1435–1450

    Google Scholar 

  15. Alaerts K, Swinnen SP, Wenderoth N (2016) Sex differences in autism: a resting-state fmri investigation of functional brain connectivity in males and females. Soc Cogn Affect Neurosci 11(6):1002–1016

    Google Scholar 

  16. Floris DL, Lai MC, Nath T, Milham MP, Di Martino A (2018) Network-specific sex differentiation of intrinsic brain function in males with autism. Mol Autism 9:17

    Google Scholar 

  17. Olson LA, Mash LE, Linke A, Fong CH, Mueller RA, Fishman I. Sex-related patterns of intrinsic functional connectivity in children and adolescents with autism spectrum disorders. Autism (in press)

  18. Smith REW, Avery JA, Wallace GL, Kenworthy L, Gotts SJ, Martin A (2019) Sex differences in resting-state functional connectivity of the cerebellum in autism spectrum disorder. Front Hum Neurosci 13:104

    Google Scholar 

  19. Di Martino A, O’Connor D, Chen B, Alaerts K, Anderson JS, Assaf M, Balsters JH, Baxter L, Beggiato A, Bernaerts S, Blanken LM, Bookheimer SY, Braden BB, Byrge L, Castellanos FX, Dapretto M, Delorme R, Fair DA, Fishman I, Fitzgerald J, Gallagher L, Keehn RJ, Kennedy DP, Lainhart JE, Luna B, Mostofsky SH, Muller RA, Nebel MB, Nigg JT, O’Hearn K, Solomon M, Toro R, Vaidya CJ, Wenderoth N, White T, Craddock RC, Lord C, Leventhal B, Milham MP (2017) Enhancing studies of the connectome in autism using the autism brain imaging data exchange II. Sci Data 4:170010

    Google Scholar 

  20. Di Martino A, Yan CG, Li Q, Denio E, Castellanos FX, Alaerts K, Anderson JS, Assaf M, Bookheimer SY, Dapretto M, Deen B, Delmonte S, Dinstein I, Ertl-Wagner B, Fair DA, Gallagher L, Kennedy DP, Keown CL, Keysers C, Lainhart JE, Lord C, Luna B, Menon V, Minshew NJ, Monk CS, Mueller S, Muller RA, Nebel MB, Nigg JT, O’Hearn K, Pelphrey KA, Peltier SJ, Rudie JD, Sunaert S, Thioux M, Tyszka JM, Uddin LQ, Verhoeven JS, Wenderoth N, Wiggins JL, Mostofsky SH, Milham MP (2014) The autism brain imaging data exchange: towards a large-scale evaluation of the intrinsic brain architecture in autism. Mol Psychiatry 19(6):659–667

    Google Scholar 

  21. Lord C, Rutter M, Goode S, Heemsbergen J, Jordan H, Mawhood L, Schopler E (1989) Autism diagnostic observation schedule: a standardized observation of communicative and social behavior. J Autism Dev Disord 19(2):185–212

    Google Scholar 

  22. Lord C, Rutter M, DiLavore PC, Risi S, Gotham K, Bishop SL (2012) Autism diagnostic observation schedule, 2nd edn. Western Psychological Services, Torrance

    Google Scholar 

  23. Lord C, Rutter M, Le Couteur A (1994) Autism diagnostic interview-revised: a revised version of a diagnostic interview for caregivers of individuals with possible pervasive developmental disorders. J Autism Dev Disord 24(5):659–685

    Google Scholar 

  24. Nair S, Jao Keehn RJ, Berkebile MM, Maximo JO, Witkowska N, Muller RA (2018) Local resting state functional connectivity in autism: site and cohort variability and the effect of eye status. Brain Imaging Behav 12(1):168–179

    Google Scholar 

  25. Power JD, Mitra A, Laumann TO, Snyder AZ, Schlaggar BL, Petersen SE (2014) Methods to detect, characterize, and remove motion artifact in resting state fmri. Neuroimage 84:320–341

    Google Scholar 

  26. Loomes R, Hull L, Mandy WPL (2017) What is the male-to-female ratio in autism spectrum disorder? A systematic review and meta-analysis. J Am Acad Child Adolesc Psychiatry 56(6):466–474

    Google Scholar 

  27. Jahedi A, Hills T, Olson M, Müller R-A, Fan J. Iterative multivariate matching package for samples with missing data: The iterMatch Package for R (under review)

  28. Cox RW (1996) Afni: software for analysis and visualization of functional magnetic resonance neuroimages. Comput Biomed Res 29(3):162–173

    Google Scholar 

  29. Smith SM, Jenkinson M, Woolrich MW, Beckmann CF, Behrens TE, Johansen-Berg H, Bannister PR, De Luca M, Drobnjak I, Flitney DE, Niazy RK, Saunders J, Vickers J, Zhang Y, De Stefano N, Brady JM, Matthews PM (2004) Advances in functional and structural mr image analysis and implementation as fsl. Neuroimage 23(Suppl 1):S208–S219

    Google Scholar 

  30. Keown CL, Datko MC, Chen CP, Maximo JO, Jahedi A, Muller RA (2017) Network organization is globally atypical in autism: a graph theory study of intrinsic functional connectivity. Biol Psychiatry Cognit Neurosci Neuroimaging 2(1):66–75

    Google Scholar 

  31. Hallquist MN, Hwang K, Luna B (2013) The nuisance of nuisance regression: spectral misspecification in a common approach to resting-state fmri preprocessing reintroduces noise and obscures functional connectivity. Neuroimage 82:208–225

    Google Scholar 

  32. Gordon EM, Laumann TO, Adeyemo B, Huckins JF, Kelley WM, Petersen SE (2016) Generation and evaluation of a cortical area parcellation from resting-state correlations. Cereb Cortex 26(1):288–303

    Google Scholar 

  33. Desikan RS, Segonne F, Fischl B, Quinn BT, Dickerson BC, Blacker D, Buckner RL, Dale AM, Maguire RP, Hyman BT, Albert MS, Killiany RJ (2006) An automated labeling system for subdividing the human cerebral cortex on mri scans into gyral based regions of interest. Neuroimage 31(3):968–980

    Google Scholar 

  34. Diedrichsen J, Balsters JH, Flavell J, Cussans E, Ramnani N (2009) A probabilistic mr atlas of the human cerebellum. Neuroimage 46(1):39–46

    Google Scholar 

  35. Hothorn T, Hornik K, Zeileis A (2006) Unbiased recursive partitioning: a conditional inference framework. J Comput Gr Stat 15(3):651–674

    MathSciNet  Google Scholar 

  36. R Core Team (2013) R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna

    Google Scholar 

  37. Liaw A, Wiener M (2002) Classification and regression by random forest. R News 2(3):18–22

    Google Scholar 

  38. Strobl C, Boulesteix AL, Kneib T, Augustin T, Zeileis A (2008) Conditional variable importance for random forests. BMC Bioinform 9:307

    Google Scholar 

  39. He Y, Byrge L, Kennedy DP (2020) Nonreplication of functional connectivity differences in autism spectrum disorder across multiple sites and denoising strategies. Hum Brain Mapp 41(5):1334–1350

    Google Scholar 

  40. Abraham A, Milham MP, Di Martino A, Craddock RC, Samaras D, Thirion B, Varoquaux G (2017) Deriving reproducible biomarkers from multi-site resting-state data: an autism-based example. Neuroimage 147:736–745

    Google Scholar 

  41. Anderson JS, Nielsen JA, Froehlich AL, DuBray MB, Druzgal TJ, Cariello AN, Cooperrider JR, Zielinski BA, Ravichandran C, Fletcher PT, Alexander AL, Bigler ED, Lange N, Lainhart JE (2011) Functional connectivity magnetic resonance imaging classification of autism. Brain 134(Pt 12):3742–3754

    Google Scholar 

  42. Kassraian-Fard P, Matthis C, Balsters JH, Maathuis MH, Wenderoth N (2016) Promises, pitfalls, and basic guidelines for applying machine learning classifiers to psychiatric imaging data, with autism as an example. Front Psychiatry 7:177

    Google Scholar 

  43. Yahata N, Morimoto J, Hashimoto R, Lisi G, Shibata K, Kawakubo Y, Kuwabara H, Kuroda M, Yamada T, Megumi F, Imamizu H, Nanez JE, Takahashi H, Okamoto Y, Kasai K, Kato N, Sasaki Y, Watanabe T, Kawato M (2016) A small number of abnormal brain connections predicts adult autism spectrum disorder. Nat Commun 7:1–2

    Google Scholar 

  44. Plitt M, Barnes KA, Martin A (2015) Functional connectivity classification of autism identifies highly predictive brain features but falls short of biomarker standards. Neuroimage Clin 7:359–366

    Google Scholar 

  45. Lombardo MV, Eyler L, Moore A, Datko M, Carter Barnes C, Cha D, Courchesne E, Pierce K (2019) Default mode-visual network hypoconnectivity in an autism subtype with pronounced social visual engagement difficulties. Elife 8:e47427

    Google Scholar 

  46. Lynn AC, Padmanabhan A, Simmonds D, Foran W, Hallquist MN, Luna B, O’Hearn K (2018) Functional connectivity differences in autism during face and car recognition: underconnectivity and atypical age-related changes. Dev Sci 21(1):e12508

    Google Scholar 

  47. Samson F, Mottron L, Soulieres I, Zeffiro TA (2012) Enhanced visual functioning in autism: an ale meta-analysis. Hum Brain Mapp 33(7):1553–1581

    Google Scholar 

  48. Reiter, M. A., Mash, L. E., Linke, A. C., Fong, C. H., Fishman, I., & Muller, R. A. (2018). Distinct Patterns of Atypical Functional Connectivity in Lower-Functioning Autism. Biol Psychiatry Cogn Neurosci Neuroimaging, 4(3), 251-259.

  49. Farrant K, Uddin LQ (2016) Atypical developmental of dorsal and ventral attention networks in autism. Dev Sci 19(4):550–563

    Google Scholar 

  50. Nomi JS, Uddin LQ (2015) Developmental changes in large-scale network connectivity in autism. Neuroimage Clin 7:732–741

    Google Scholar 

Download references

Acknowledgments

This study was supported by the National Institutes of Health R01 MH081023 (RAM) and R01 MH101173 (RAM). Authors MAR and AJ were supported by San Diego State University Graduate Fellowships. MAR was also supported by the Autism Speaks Sir Dennis Weatherstone Fellowship program (Grant No. 10609). ARJF was supported by the Science Engineering and Research Board (SERB)/Indo-Us Science and Technology Forum (IUSSTF), Grant 119. We would like to acknowledge here the participants and families who enrolled in the current study, without whom the current project would not have been possible.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Ralph-Axel Müller.

Ethics declarations

Conflicts of interest

The authors have no known conflicts of interest.

Ethics approval

All data collection and sharing were conducted under the approval of the appropriate Institutional Review Boards.

Consent to participate

All participants provided written consent to participate and share their de-identified data for the purposes of research.

Consent for publication

All authors were involved in the preparation of this manuscript and consent to its publication.

Availability of data and material

All of the data included in the current study is publically available through the Autism Brain Imaging Data Exchange (ABIDE) initiative online.

Code availability

A combination of publically available neuroimaging processing software, statistical analysis software, and custom code generated by the research team was used in the preparation of this manuscript.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (DOCX 631 kb)

Rights and permissions

Reprints and Permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Reiter, M.A., Jahedi, A., Fredo, A.R.J. et al. Performance of machine learning classification models of autism using resting-state fMRI is contingent on sample heterogeneity. Neural Comput & Applic 33, 3299–3310 (2021). https://doi.org/10.1007/s00521-020-05193-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-020-05193-y

Keywords

Navigation