Abstract
We comprehensively compared all available questionnaires for measuring quantitative autistic traits (QATs) in terms of reliability and construct validity in 3,147 non-clinical and 60 clinical subjects with normal intelligence. We examined four full-length forms, the Subthreshold Autism Trait Questionnaire (SATQ), the Broader Autism Phenotype Questionnaire, the Social Responsiveness Scale2-Adult Self report (SRS2-AS), and the Autism-Spectrum Quotient (AQ). The SRS2-AS and the AQ each had several short forms that we also examined, bringing the total to 11 forms. Though all QAT questionnaires showed acceptable levels of test–retest reliability, the AQ and SRS2-AS, including their short forms, exhibited poor internal consistency and discriminant validity, respectively. The SATQ excelled in terms of classical test theory and due to its short length.
Similar content being viewed by others
References
Allison, C., Auyeung, B., & Baron-Cohen, S. (2012). Toward brief “Red Flags” for autism screening: The Short Autism Spectrum Quotient and the Short Quantitative Checklist for Autism in toddlers in 1,000 cases and 3,000 controls [corrected]. Journal of the American Academy of Child and Adolescent Psychiatry, 51(2), 202–212.
Austin, E. J. (2005). Personality correlates of the broader autism phenotype as assessed by the Autism Spectrum Quotient (AQ). Personality and Individual Differences, 38(2), 451–460.
Bailey, A., Palferman, S., Heavey, L., & Le Couteur, A. (1998). Autism: The phenotype in relatives. Journal of Autism and Developmental Disorders, 28(5), 369–392.
Baron-Cohen, S., Wheelwright, S., Skinner, R., Martin, J., & Clubley, E. (2001). The autism-spectrum quotient (AQ): Evidence from Asperger syndrome/high-functioning autism, males and females, scientists and mathematicians. Journal of Autism and Developmental Disorders, 31(1), 5–17.
Bolton, P., Macdonald, H., Pickles, A., Rios, P., Goode, S., Crowson, M., et al. (1994). A case–control family history study of autism. Journal of Child Psychology and Psychiatry Child Psychology and Psychiatry, and Allied Disciplines, 35(5), 877–900.
Boulet, J., & Boss, M. W. (1991). Reliability and validity of the brief symptom inventory. Psychological Assessment, 3(3), 433–437.
Brugha, T. S., McManus, S., Smith, J., Scott, F. J., Meltzer, H., Purdon, S., et al. (2012). Validating two survey methods for identifying cases of autism spectrum disorder among adults in the community. Psychological Medicine, 42(3), 647–656.
Campbell, D. T., & Fiske, D. W. (1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56(2), 81–105.
Canty, A., & Ripley, B. (2012). boot: Bootstrap R (S-Plus) functions.
Constantino, J. N., Abbacchi, A. M., Lavesser, P. D., Reed, H., Givens, L., Chiang, L., et al. (2009). Developmental course of autistic social impairment in males. Development and Psychopathology, 21(1), 127–138.
Constantino, J. N., & Cruber, C. P. (2012). Social Responsiveness Scale, second edition (SRS-2). Los Angeles, CA: Western Psychological Services.
Constantino, J. N., Lavesser, P. D., Zhang, Y., Abbacchi, A. M., Gray, T., & Todd, R. D. (2007). Rapid quantitative assessment of autistic social impairment by classroom teachers. Journal of the American Academy of Child and Adolescent Psychiatry, 46(12), 1668–1676.
Constantino, J. N., & Todd, R. D. (2003). Autistic traits in the general population: A twin study. Archives of General Psychiatry, 60(5), 524–530.
Davidson, J., Goin-Kochel, R. P., Green-Snyder, L. A., Hundley, R. J., Warren, Z., & Peters, S. U. (2012). Expression of the broad autism phenotype in simplex autism families from the simons simplex collection. Journal of Autism and Developmental Disorders. doi:10.1007/s10803-012-1492-1.
DeLong, E. R., DeLong, D. M., & Clarke-Pearson, D. L. (1988). Comparing the areas under two or more correlated receiver operating characteristic curves: A nonparametric approach. Biometrics, 44(3), 837–845.
Derogatis, L. R. (1977). SCL-90R (revised version) manual I. Baltimore: Johns Hopkins University School of Medicine.
Derogatis, L. R., & Melisaratos, N. (1983). The Brief Symptom Inventory: An introductory report. Psychological Medicine, 13(03), 595–605.
Duku, E., Vaillancourt, T., Szatmari, P., Georgiades, S., Zwaigenbaum, L., Smith, I. M., et al. (2013). Investigating the measurement properties of the social responsiveness scale in preschool children with autism spectrum disorders. Journal of Autism and Developmental Disorders, 43(4), 860–868.
Efron, B., & Tibshirani, R. (1994). An introduction to the bootstrap (Chapman & Hall/CRC monographs on statistics & applied probability) (1st ed.). New York: Chapman and Hall/CRC.
Fleiss, J. L. (1999). The design and analysis of clinical experiments. Hoboken, NJ: Wiley.
Folstein, S., & Rutter, M. (1977). Infantile autism: A genetic study of 21 twin pairs. Journal of Child Psychology and Psychiatry Child Psychology and Psychiatry, and Allied Disciplines, 18(4), 297–321.
Froom, P., Melamed, S., Kristal-Boneh, E., Benbassat, J., & Ribak, J. (1999). Healthy volunteer effect in industrial workers. Journal of Clinical Epidemiology, 52(8), 731–735.
Fujita, K., Maekawa, H., Dairoku, K., & Yamanaka, K. (2006). A Japanese version of the WAIS-III. Tokyo: Nihon Bunka Kagakusha. Tokyo: Nihon Bunka Kagakusha (in Japanese).
Fukuhara, S., Bito, S., Green, J., Hsiao, A., & Kurokawa, K. (1998). Translation, adaptation, and validation of the SF-36 Health Survey for use in Japan. Journal of Clinical Epidemiology, 51(11), 1037–1044.
Fukuhara, S., & Suzukamo, Y. (2004). Manual of the SF-36v2 Japanese version: Institute for Health Outcomes and Process Evaluation research. Kyoto: Institute for Health Outcomes and Process Evaluation Research.
Furukawa, T. A., Kawakami, N., Saitoh, M., Ono, Y., Nakane, Y., Nakamura, Y., et al. (2008). The performance of the Japanese version of the K6 and K10 in the World Mental Health Survey Japan. International Journal of Methods in Psychiatric Research, 17(3), 152–158.
Gamer, M., Lemon, J., Fellows, I., & Singh, R. (2010). irr: Various coefficients of interrater reliability and agreement for quantative, ordinal, and nominal data. R package version 1.4-9.
Hoekstra, R. A., Bartels, M., Cath, D. C., & Boomsma, D. I. (2008). Factor structure, reliability and criterion validity of the Autism-Spectrum Quotient (AQ): A study in Dutch population and patient groups. Journal of Autism and Developmental Disorders, 38(8), 1555–1566.
Hoekstra, R. A., Vinkhuyzen, A. A. E., Wheelwright, S., Bartels, M., Boomsma, D. I., Baron-Cohen, S., et al. (2011). The construction and validation of an abridged version of the autism-spectrum quotient (AQ-Short). Journal of Autism and Developmental Disorders, 41(5), 589–596.
Hurley, R. S. E., Losh, M., Parlier, M., Reznick, J. S., & Piven, J. (2007). The broad autism phenotype questionnaire. Journal of Autism and Developmental Disorders, 37(9), 1679–1690.
Hurst, R. M., Nelson-Gray, R. O., Mitchell, J. T., & Kwapil, T. R. (2007). The relationship of Asperger’s characteristics and schizotypal personality traits in a non-clinical adult sample. Journal of Autism and Developmental Disorders, 37(9), 1711–1720.
Ingersoll, B., Hopwood, C. J., Wainer, A., & Brent Donnellan, M. (2011). A comparison of three self-report measures of the broader autism phenotype in a non-clinical sample. Journal of Autism and Developmental Disorders, 41(12), 1646–1657.
Japanese version of WISC III and Publication Committee. (1998). A Japanese version of WISC III. Tokyo: Nihon Bunka Kagakusha. (in Japanese).
Johns, M. W. (1991). A new method for measuring daytime sleepiness: The Epworth sleepiness scale. Sleep, 14(6), 540–545.
Kanne, S. M., Christ, S. E., & Reiersen, A. M. (2009). Psychiatric symptoms and psychosocial difficulties in young adults with autistic traits. Journal of Autism and Developmental Disorders, 39(6), 827–833.
Kanne, S. M., Wang, J., & Christ, S. E. (2012). The Subthreshold Autism Trait Questionnaire (SATQ): Development of a brief self-report measure of subthreshold autism traits. Journal of Autism and Developmental Disorders, 42(5), 769–780.
Kanner, L. (1943). Autistic disturbances of affective contact. Nervous Child, 2, 217–250.
Kawakami, N., Arai, S., & Okoyama, K. (1993). Reliability and validity of the Japanese version of the CAGE alcoholism screening test. Japanese Journal of Hygiene, 48(1), 401.
Kessler, R. C., Andrews, G., Colpe, L. J., Hiripi, E., Mroczek, D. K., Normand, S. L. T., et al. (2002). Short screening scales to monitor population prevalences and trends in non-specific psychological distress. Psychological Medicine, 32(6), 959–976.
Kirshner, B., & Guyatt, G. (1985). A methodological framework for assessing health indices. Journal of Chronic Diseases, 38(1), 27–36.
Kloosterman, P. H., Keefer, K. V., Kelley, E. A., Summerfeldt, L. J., & Parker, J. D. A. (2011). Evaluation of the factor structure of the Autism-Spectrum Quotient. Personality and Individual Differences, 50(2), 310–314.
Kurita, H., Koyama, T., & Osada, H. (2005). Autism-Spectrum Quotient-Japanese version and its short forms for screening normally intelligent persons with pervasive developmental disorders. Psychiatry and Clinical Neurosciences, 59(4), 490–496.
Lau, W. Y.-P., Gau, S. S.-F., Chiu, Y.-N., Wu, Y.-Y., Chou, W.-J., Liu, S.-K., et al. (2013). Psychometric properties of the Chinese version of the Autism Spectrum Quotient (AQ). Research in Developmental Disabilities, 34(1), 294–305.
Lord, C., Risi, S., Lambrecht, L., Cook, E. H., Leventhal, B. L., DiLavore, P. C., et al. (2000). The autism diagnostic observation schedule-generic: A standard measure of social and communication deficits associated with the spectrum of autism. Journal of Autism and Developmental Disorders, 30(3), 205–223.
Lord, C., Rutter, M., & Le Couteur, A. (1994). Autism Diagnostic Interview-Revised: A revised version of a diagnostic interview for caregivers of individuals with possible pervasive developmental disorders. Journal of Autism and Developmental Disorders, 24(5), 659–685.
Mandy, W. P. L., Charman, T., & Skuse, D. H. (2012). Testing the construct validity of proposed criteria for DSM-5 autism spectrum disorder. Journal of the American Academy of Child and Adolescent Psychiatry, 51(1), 41–50.
Mandy, W. P. L., & Skuse, D. H. (2008). Research review: What is the association between the social-communication element of autism and repetitive interests, behaviours and activities? Journal of Child Psychology and Psychiatry, 49(8), 795–808.
Mayfield, D., McLeod, G., & Hall, P. (1974). The CAGE questionnaire: Validation of a new alcoholism screening instrument. The American Journal of Psychiatry, 131(10), 1121–1123.
McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30–46.
McHorney, C. A., Ware, J. E., Lu, J. F., & Sherbourne, C. D. (1994). The MOS 36-item Short-Form Health Survey (SF-36): III. Tests of data quality, scaling assumptions, and reliability across diverse patient groups. Medical Care, 32(1), 40–66.
Messick, S. (1995). Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. American Psychologist, 50(9), 741–749.
Micali, N., Chakrabarti, S., & Fombonne, E. (2004). The broad autism phenotype: Findings from an epidemiological survey. Autism, 8(1), 21–37.
Nunnally, J., & Bernstein, I. (1994). Psychometric theory. rds.epi-ucsf.org. New York: Nunnally, Jum C., and Ira H. Bernstein. Psychometric theory. No. 972. McGraw-Hill Humanities/Social Sciences/Languages, 1994.
Piersma, H. L., Boes, J. L., & Reaume, W. M. (1994). Unidimensionality of the Brief Symptom Inventory (BSI) in adult and adolescent inpatients. Journal of Personality Assessment, 63(2), 338–344.
Piven, J., Gayle, J., Chase, G. A., Fink, B., Landa, R., Wzorek, M. M., et al. (1990). A family history study of neuropsychiatric disorders in the adult siblings of autistic individuals. Journal of the American Academy of Child and Adolescent Psychiatry, 29(2), 177–183.
Piven, J., Palmer, P., Jacobi, D., Childress, D., & Arndt, S. (1997). Broader autism phenotype: Evidence from a family history study of multiple-incidence autism families. The American Journal of Psychiatry, 154(2), 185–190.
Podsakoff, P. M., MacKenzie, S. B., Lee, J.-Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. The Journal of Applied Psychology, 88(5), 879–903.
R Development CoreTeam. (2012). R: A language and environment for statistical computing. Vienna: R Foundation for Statistical Computing.
Ransohoff, D. F., & Feinstein, A. R. (1978). Problems of spectrum and bias in evaluating the efficacy of diagnostic tests. The New England Journal of Medicine, 299(17), 926–930.
Reiersen, A. M., Constantino, J. N., Grimmer, M., Martin, N. G., & Todd, R. D. (2008). Evidence for shared genetic influences on self-reported ADHD and autistic symptoms in young adult Australian twins. Twin Research and Human Genetics, 11(6), 579–585.
Rizopoulos, D. (2006). ltm: An R package for latent variable modeling and item response analysis. Journal of Statistical Software, 17(5), 1–25.
Robin, X., Turck, N., Hainard, A., Tiberti, N., Lisacek, F., Sanchez, J.-C., et al. (2011). pROC: An open-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics, 12, 77.
Ruta, L., Mazzone, D., Mazzone, L., Wheelwright, S., & Baron-Cohen, S. (2012). The Autism-Spectrum Quotient—Italian version: A cross-cultural confirmation of the broader autism phenotype. Journal of Autism and Developmental Disorders, 42(4), 625–633.
Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428.
St Pourcain, B., Wang, K., Glessner, J. T., Golding, J., Steer, C., Ring, S. M., et al. (2010). Association between a high-risk autism locus on 5p14 and social communication spectrum phenotypes in the general population. The American journal of psychiatry, 167(11), 1364–1372.
Steiger, J. H. (1980). Tests for comparing elements of a correlation matrix. Psychological Bulletin, 82(2), 245.
Streiner, D. L., & Norman, G. R. (2008). Health Measurement Scales: A practical guide to their development and use. New York, NY: Oxford University Press.
Takegami, M., Suzukamo, Y., Wakita, T., Noguchi, H., Chin, K., Kadotani, H., et al. (2009). Development of a Japanese version of the Epworth Sleepiness Scale (JESS) based on item response theory. Sleep Medicine, 10(5), 556–565.
Tanaka Institute of Education. (1987). Tanaka-Binet Intelligence Scale. Tokyo: Taken. (in Japanese).
Tomioka, M., Shimura, M., Hidaka, M., & Kubo, C. (2008). The reliability and validity of a Japanese version of symptom checklist 90 revised. BioPsychoSocial Medicine, 2, 19.
Wakabayashi, A., Baron-Cohen, S., & Wheelwright, S. (2006a). Are autistic traits an independent personality dimension? A study of the Autism-Spectrum Quotient (AQ) and the NEO-PI-R. Personality and Individual Differences, 41(5), 873–883.
Wakabayashi, A., Baron-Cohen, S., Wheelwright, S., & Tojo, Y. (2006b). The Autism-Spectrum Quotient (AQ) in Japan: A cross-cultural comparison. Journal of Autism and Developmental Disorders, 36(2), 263–270.
Ware, J. E., & Gandek, B. (1998). Methods for testing data quality, scaling assumptions, and reliability. Journal of Clinical Epidemiology, 51(11), 945–952.
Ware, J. E., Kosinski, M., & Dewey, J. E. (2000). How to score version 2 of the SF-36 Health Survey. Lincoln, RI: QualityMetric Incorporated.
Watson, D., & Clark, L. A. (1984). Negative affectivity: The disposition to experience aversive emotional states. Psychological Bulletin, 96(3), 465–490.
Wild, D., Grove, A., Martin, M., Eremenco, S., McElroy, S., Verjee-Lorenz, A., et al. (2005). Principles of good practice for the translation and cultural adaptation process for patient-reported outcomes (PRO) measures: Report of the ISPOR task force for translation and cultural adaptation. Value in Health, 8(2), 94–104.
Woodbury-Smith, M. R., Robinson, J., Wheelwright, S., & Baron-Cohen, S. (2005). Screening adults for Asperger Syndrome using the AQ: A preliminary study of its diagnostic validity in clinical practice. Journal of Autism and Developmental Disorders, 35(3), 331–335.
Acknowledgments
This study was supported by the Grants-in-Aid for Scientific Research (Basic Research B: No. 22300095, Basic Research C: No. 24590807 and No. 23591723) from the Ministry of Education, Culture, Sports, Science and Technology in Japan. The authors would like to thank the following people for their assistance with data collection: Dr. Taishi Miyachi, Dr. Masayuki Imaeda, Dr. Atsuro Yamada, Dr. Takao Maeda, Dr. Susumu Adachi, Dr. Hiroshi Hirai, and Dr. Chiho Goto. They also thank Prof. John N. Constantino and Dr. Christian P. Gruber for their useful and constructive comments.
Author information
Authors and Affiliations
Corresponding author
Electronic supplementary material
Below is the link to the electronic supplementary material.
10803_2013_2020_MOESM13_ESM.tif
Distribution and Q-Q plots of scores from the SRS2-AS short forms according to disease status. (Supplementary Figure 1, TIFF 15005 kb)
10803_2013_2020_MOESM14_ESM.tif
Distribution and Q-Q plot of scores from the AQ short forms according to disease status. (Supplementary Figure 2, TIFF 39461 kb)
10803_2013_2020_MOESM15_ESM.tif
ROC curves for the four full-scale questionnaires of autistic traits to discriminate ASD patients from non-ASD controls. (Supplementary Figure 3, TIFF 39 kb)
10803_2013_2020_MOESM16_ESM.tif
ROC curves for the SRS2-AS and its abridged forms to discriminate ASD patients from non-ASD controls. (Supplementary Figure 4, TIFF 36 kb)
10803_2013_2020_MOESM17_ESM.tif
ROC curves for the AQ and its abridged forms to discriminate ASD patients from non-ASD controls. (Supplementary Figure 5, TIFF 43 kb)
Rights and permissions
About this article
Cite this article
Nishiyama, T., Suzuki, M., Adachi, K. et al. Comprehensive Comparison of Self-administered Questionnaires for Measuring Quantitative Autistic Traits in Adults. J Autism Dev Disord 44, 993–1007 (2014). https://doi.org/10.1007/s10803-013-2020-7
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10803-013-2020-7