Empirical Evaluation of the Generalizability of the Sample from the Multimodal Treatment Study for ADHD

  • Jack Stevens
  • Kelly Kelleher
  • Joel Greenhouse
  • Guanmin Chen
  • Huiyun Xiang
  • Eloise Kaizar
  • Peter S. Jensen
  • L. Eugene Arnold
Original Paper

Abstract

Randomized controlled trials (RCTs) are criticized for the questionable generalizability of their patient samples to real world populations. This study compared the demographic and clinical characteristics of children from the Multimodal Treatment Study of Children with ADHD (MTA) to those from the NIMH Methods for the Epidemiology of Child and Adolescent Mental Disorders (MECA) Study and the National Health and Nutrition Examination Survey (NHANES). On the whole, we found no compelling evidence against the representativeness of the MTA sample, but the limited power from MECA and NHANES may have lead to noteworthy differences between datasets being statistically nonsignificant.

Keywords

ADHD External validity Generalizability 

References

  1. American Psychiatric Association. (1994). Diagnostic and statistical manual of mental disorders (4th ed.). Washington, DC: American Psychiatric Association.Google Scholar
  2. Arnold, L. E., Abikoff, H. B., Cantwell, D. P., Conners, C. K., Elliott, G., Greenhill, L. L., Hechtman, L., Hinshaw, S. P., Hoza, B., Jensen, P. S., Kraemer, H. C., March, J. S., Newcorn, J. H., Pelham, W. E., Richters, J. E., Schiller, E., Severe, J. B., Swanson, J. M., Vereen, D., & Wells, K. C. (1997). National Institute of Mental Health Collaborative Multimodal Treatment Study of Children with ADHD (the MTA). Design challenges and choices. Archives of General Psychiatry, 54, 865–870.PubMedGoogle Scholar
  3. Arnold, L. E., Elliott, M., Sachs, L., Bird, H., Kraemer, H. C., Wells, K. C., Abikoff, H. B., Comarda, A., Conners, C. K., Elliott, G. R., Greenhill, L. L., Hechtman, L., Hinshaw, S. P., Hoza, B., Jensen, P. J., March, J. S., Newcorn, J., Pelham, W. E., Severe, J. B., Swanson, J. M., Vitiello, B., & Wigal, T. (2003). Effects of ethnicity on treatment attendance, stimulant response/dose, and 14-month outcome in ADHD. Journal of Consulting and Clinical Psychology, 71, 713–727.PubMedCrossRefGoogle Scholar
  4. Barkley, R. A. (2000). Commentary on the multimodal treatment study of children with ADHD. Journal of Abnormal Child Psychology, 28, 595–599.PubMedCrossRefGoogle Scholar
  5. Beck, A. T., Ward, C. H., Mendelson, M., Mock, J. E., & Erbaugh, J. K. (1961). An inventory for measuring depression. Archives of General Psychiatry, 4, 561–571.PubMedGoogle Scholar
  6. Bird, H. R., Gould, M. S., & Staghezza, B. M. (1993). Patterns of diagnostic comorbidity in a community sample of children aged 9 through 16 years. Journal of the American Academy of Child and Adolescent Psychiatry, 32, 361–368.PubMedCrossRefGoogle Scholar
  7. Conners, C. K., Epstein, J. N., March, J. S., Angold, A., Wells, K. C., Klaric, J., Swanson, J. M., Arnold, L. E., Abikoff, H. B., Elliott, G. R., Greenhill, L. L., Hechtman, L., Hinshaw, S. P., Hoza, B., Jensen, P. S., Kraemer, H. C., Newcorn, J. H., Pelham, W. E., Severe, J. B., Vitiello, B., & Wigal, T. (2001). Multimodal treatment of ADHD in the MTA: An alternative outcome analysis. Journal of the American Academy of Child and Adolescent Psychiatry, 40, 159–167.PubMedCrossRefGoogle Scholar
  8. Dunn, L. M., Dunn, L. M. (1981). Peabody picture vocabulary test-revised. Circle Pines, MN: American Guidance Service.Google Scholar
  9. Farmer, E. M., Stangl, D. K., Burns, B. J., Costello, E. J., & Angold, A. (1999). Use, persistence, and intensity: Patterns of care for children’s mental health across one year. Community Mental Health Journal, 35, 31–46.PubMedCrossRefGoogle Scholar
  10. Fisher, P., Wicks, J., Shaffer, D., Piacentini, J., & Lapkin, J. (1992). NIMH diagnostic interview schedule for children, version 2.3, user’s manual. New York: New York State Psychiatric Institute.Google Scholar
  11. General Accounting Office. (1992). Cross design synthesis: A new strategy for medical effectiveness research. Washington, DC: GAO.Google Scholar
  12. Greenhill, L. L., Swanson, J. M., Vitiello, B., Davies, M., Clevenger, W., Wu, M., Arnold, L. E., Abikoff, H. B., Bukstein, O. G., Conners, C. K., Elliott, G. R., Hechtman, L., Hinshaw, S. P., Hoza, B., Jensen, P. S., Kraemer, H. C., March, J. S., Newcorn, J. H., Severe, J. B., Wells, K., & Wigal, T. (2001). Impairment and deportment responses to different methylphenidate doses in children with ADHD: The MTA titration trial. Journal of the American Academy of Child and Adolescent Psychiatry, 40, 180–187.PubMedCrossRefGoogle Scholar
  13. Jensen, P. S., Hinshaw, S. P., Kraemer, H. C., Lenora, N., Newcorn, J. H., Abikoff, H. B., March, J. S., Arnold, L. E., Cantwell, D. P., Conners, C. K., Elliott, G. R., Greenhill, L. L., Hechtman, L., Hoza, B., Pelham, W. E., Severe, J. B., Swanson, J. M., Wells, K. C., Wigal, T., & Vitiello, B. (2001). ADHD comorbidity findings from the MTA study: Comparing comorbid subgroups. Journal of the American Academy of Child and Adolescent Psychiatry, 40, 147–158.PubMedCrossRefGoogle Scholar
  14. Korn, E. L., & Graubard, B. I. (1999). Analysis of health surveys. New York: Wiley Press.Google Scholar
  15. Lahey, B. B., Flagg, E. W., Bird, H. R., Schwab-Stone, M. E., Canino, G., Dulcan, M. K., Leaf, P. J., Davies, M., Brogan, D., Bourdon, K., Horwitz, S. M., Rubio-Stipec, M., Freeman, D. H., Lichtman, J. H., Shaffer, D., Goodman, S. H., Narrow, W. E., Weissman, M. M., Kandel, D. B., Jensen, P. S., Richters, J. E., & Regier, D. A. (1996). The NIMH Methods for the Epidemiology of Child and Adolescent Mental Disorders (MECA) study: Background and methodology. Journal of the American Academy of Child and Adolescent Psychiatry, 35, 855–864.PubMedCrossRefGoogle Scholar
  16. Leaf, P., Alegria, M., Cohen, P., Goodman, S., Horwitz, S. M., Hoven, C. W., Narrow, W. E., Vaden-Kiernan, M., & Regier, D. A. (1996). Mental health service use in the community and schools: Results from the four-community MECA study. Journal of the American Academy of Child and Adolescent Psychiatry, 35, 889–897.PubMedCrossRefGoogle Scholar
  17. MTA Cooperative Group. (1999a). A 14-month randomized clinical trial of treatment strategies for attention-deficit/hyperactivity disorder. Multimodal Treatment Study of Children with ADHD. Archives of General Psychiatry, 56, 1073–1086.CrossRefGoogle Scholar
  18. MTA Cooperative Group. (1999b). Moderators and mediators of treatment response for children with attention-deficit/hyperactivity disorder: The Multimodal Treatment Study of children with Attention-deficit/hyperactivity disorder. Archives of General Psychiatry, 56, 1088–1096.CrossRefGoogle Scholar
  19. Owens, E. B., Hinshaw, S. P., Kraemer, H. C., Arnold, L. E., Abikoff, H. B., Cantwell, D. P., Conners, C. K., Elliott, G., Greenhill, L. L., Hechtman, L., Hoza, B., Jensen, P. S., March, J. S., Newcorn, J. H., Pelham, W. E., Severe, J. B., Swanson, J. M., Vitiello, B., Wells, K. C., & Wigal, T. (2003). Which treatment for whom for ADHD? Moderators of treatment response in the MTA. Journal of Consulting and Clinical Psychiatry, 71, 540–552.CrossRefGoogle Scholar
  20. Rieppi, R., Greenhill, L. L., Ford, R. E., Chuang, S., Wu, M., Davies, M., Abikoff, H. B., Arnold, L. E., Conners, C. K., Elliott, G. R., Hechtman, L., Hinshaw, S. P., Hoza, B., Jensen, P. S., Kraemer, H. C., March, J. S., Newcorn, J. H., Pelham, W. E., Severe, J. B., Swanson, J. M., Vitiello, B., Wells, K. C., & Wigal, T. (2002). Socioeconomic status as a moderator of ADHD treatment outcomes. Journal of the American Academy of Child and Adolescent Psychiatry, 41, 269–277.PubMedCrossRefGoogle Scholar
  21. Rothwell, P. (2005). External validity of randomised controlled trials: To whom do the results of this trial apply? Lancet, 365, 82–93.PubMedCrossRefGoogle Scholar
  22. Shaffer, D., Fisher, P., Dulcan, M. K., Davies, M., Piacentini, J., Schwab-Stone, M. E., Lahey, B. B., Bourdon, K., Jensen, P. S., Bird, H. R., Canino, G., & Regier, D. A. (1996). The NIMH Diagnostic Interview Schedule for Children Version 2.3 (DISC-2.3): Description, acceptability, prevalence rates, and performance in the MECA Study. Methods for the Epidemiology of Child and Adolescent Mental Disorders Study. Journal of the American Academy of Child and Adolescent Psychiatry, 35, 865–877.PubMedCrossRefGoogle Scholar
  23. Persons, J. B., & Silberschatz, G. (1998). Are results of randomized controlled trials useful to psychotherapists? Journal of Consulting and Clinical Psychology, 66, 126–135.PubMedCrossRefGoogle Scholar
  24. SPSS Inc. (2003). SPSS base 12.0 application guide. Chicago, IL: Author.Google Scholar
  25. Stevens, J., Harman, J. S., & Kelleher, K. J. (2005). Race/ethnicity and insurance status as factors associated with ADHD treatment patterns. Journal of Child and Adolescent Psychopharmacology, 15, 88–96.PubMedCrossRefGoogle Scholar
  26. Stirman, S. W., DeRubeis, R. J., Crits-Christoph, P., & Brody, P. E. (2003). Are samples in randomized controlled trials of psychotherapy representative of community outpatients? A new methodology and initial findings. Journal of Consulting and Clinical Psychology, 71, 963–972.PubMedCrossRefGoogle Scholar
  27. Swanson, J. M., Kraemer, H. C., Hinshaw, S. P., Arnold, L. E., Conners, C. K., Abikoff, H. B., Clevenger, W., Davies, M., Elliott, G. R., Greenhill, L. L., Hechtman, L., Hoza, B., Jensen, P. S., March, J. S., Newcorn, J. H., Owens, E. B., Pelham, W. E., Schiller, E., Severe, J. B., Simpson, S., Vitiello, B., Wells, K., Wigal, T., & Wu, M. (2001). Clinical relevance of the primary findings of the MTA: Success rates based on severity of ADHD and ODD symptoms at the end of treatment. Journal of the American Academy of Child and Adolescent Psychiatry, 40, 168–179.PubMedCrossRefGoogle Scholar
  28. Wechsler, D. (1991). Wechsler intelligence Scale for Children—Third Edition San Antonio. TX: Psychological Corporation.Google Scholar
  29. Westen, D. I. (2006). Patients and treatments in clinical trials are not adequately representative of clinical practice. In J. C. Norcross, L. E. Beutler, & R. F. Levant (Eds.), Evidence-based practices in mental health: Debate and dialogue on the fundamental questions. Washington, DC: American Psychological Association.Google Scholar
  30. Zimmerman, M., Chelminski, I., & Posternak, M. A. (2005). Generalizability of antidepressant efficacy trials: Differences between depressed psychiatric outpatients who would versus would not qualify for an efficacy trial. American Journal of Psychiatry, 162, 1370–1372.PubMedCrossRefGoogle Scholar
  31. Zimmerman, M., Mattia, J. I., & Posternak, M. A. (2002). Are subjects in pharmacological treatment trials of depression representative of patients in routine clinical practice? American Journal of Psychiatry, 159, 469–473.PubMedCrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2006

Authors and Affiliations

  • Jack Stevens
    • 1
  • Kelly Kelleher
    • 1
  • Joel Greenhouse
    • 2
  • Guanmin Chen
    • 3
  • Huiyun Xiang
    • 1
  • Eloise Kaizar
    • 2
  • Peter S. Jensen
    • 4
  • L. Eugene Arnold
    • 5
  1. 1.Department of PediatricsThe Ohio State UniversityColumbusUSA
  2. 2.Department of StatisticsCarnegie Mellon UniversityPittsburghUSA
  3. 3.School of Public HealthWuhan UniversityWuhanP.R. China
  4. 4.Department of Psychiatry, Center for the Advancement of Children’s Mental HealthColumbia UniversityNew YorkUSA
  5. 5.Department of PsychiatryThe Ohio State UniversityColumbusUSA

Personalised recommendations