Prevention Science

, Volume 19, Supplement 1, pp 60–73 | Cite as

Addressing Methodologic Challenges and Minimizing Threats to Validity in Synthesizing Findings from Individual-Level Data Across Longitudinal Randomized Trials

  • Ahnalee Brincks
  • Samantha Montag
  • George W. Howe
  • Shi Huang
  • Juned Siddique
  • Soyeon Ahn
  • Irwin N. Sandler
  • Hilda Pantin
  • C. Hendricks BrownEmail author


Integrative Data Analysis (IDA) encompasses a collection of methods for data synthesis that pools participant-level data across multiple studies. Compared with single-study analyses, IDA provides larger sample sizes, better representation of participant characteristics, and often increased statistical power. Many of the methods currently available for IDA have focused on examining developmental changes using longitudinal observational studies employing different measures across time and study. However, IDA can also be useful in synthesizing across multiple randomized clinical trials to improve our understanding of the comprehensive effectiveness of interventions, as well as mediators and moderators of those effects. The pooling of data from randomized clinical trials presents a number of methodological challenges, and we discuss ways to examine potential threats to internal and external validity. Using as an illustration a synthesis of 19 randomized clinical trials on the prevention of adolescent depression, we articulate IDA methods that can be used to minimize threats to internal validity, including (1) heterogeneity in the outcome measures across trials, (2) heterogeneity in the follow-up assessments across trials, (3) heterogeneity in the sample characteristics across trials, (4) heterogeneity in the comparison conditions across trials, and (5) heterogeneity in the impact trajectories. We also demonstrate a technique for minimizing threats to external validity in synthesis analysis that may result from non-availability of some trial datasets. The proposed methods rely heavily on latent variable modeling extensions of the latent growth curve model, as well as missing data procedures. The goal is to provide strategies for researchers considering IDA.


Integrative data analysis Synthesis methodology Participant-level meta-analysis Harmonization 



We gratefully acknowledge the National Institute of Mental Health Collaborative Data Synthesis for Adolescent Depression Trials Study Team, composed of our many colleagues who generously provided their data to be used in this study, obtained access to key datasets, reviewed coding decisions, provided substantive or methodologic recommendations, and reviewed this manuscript. We also thank NIMH for their support through grant number R01-MH040859 (Collaborative Data Synthesis for Adolescent Depression Trials, Brown PI) and NCATS for their support through grant number UL1-TR000460 (Miami Clinical and Translational Science Institute, Szapocznik PI), as well as the following grants that financially supported the trials: NIMH R01-MH048696 (Beardslee), NIMH R01-MH064717 (Beardslee & Gladstone), NIMH R01-MH064503 (Brent), NIMH R03-MH048118 (Clarke), NIDA R01-DA07031 (Dishion), NIDA R01-DA13773 (Dishion), NIMH R01-MH064735 (Garber), WT Grant Foundation 961730 (Garber), NIMH R01-MH064707 (Gonzales), NIDA R01-DA017462 (Pantin), NIDA R01-DA025894 (Prado), CDC U01-PS000671 (Prado), NIMH P30-MH068685 (Sandler), NIMH R01-MH049155 (Sandler & Ayers), NIMH R01-MH052270 (Seligman), NIDA R01-DA007029 (Spoth), NIDA R01-DA018374 (Stormshak), NIMH MH063042 (Szapocznik), NIMH K08-MH072918 (Van Voorhees), NIMH K23-MH071320 (Young), and NARSAD Young Investigator Grant (Young). The content of this paper is solely the responsibility of the authors and does not necessarily represent the official views of the funding agencies nor that of our collaborators who provided access to their data.

Compliance with Ethical Standards

Conflict of Interest

CHB was supported as a consultant on one of these projects (New Beginnings) and received funding support on the Familias Unidas trials as did Brincks, Huang, and Pantin who developed and directed much of research on this intervention. Sandler developed and directed much of the work on two trials in this synthesis.

Ethics Approval

This article does not contain any studies with animals performed by any of the authors. This project involving sharing of data was review by two IRBs. Use of deidentified data in this synthesis project was approved by the University of Miami and Northwestern IRBs, and all institutions signed data use agreements with Northwestern.

Informed Consent

All trials included in this synthesis were approved by their respective institutional review boards. Informed consent was obtained from all individual participants included in these respective studies.

Supplementary material

11121_2017_769_MOESM1_ESM.docx (198 kb)
Suppl Fig. A (DOCX 197 kb)
11121_2017_769_MOESM2_ESM.docx (25 kb)
Suppl Fig. B (DOCX 25 kb)
11121_2017_769_MOESM3_ESM.docx (20 kb)
Suppl Fig. C (DOCX 20 kb)
11121_2017_769_MOESM4_ESM.docx (16 kb)
Suppl Table A (DOCX 16 kb)


  1. Barch, D. M., Gotlib, I. H., Bilder, R. M., Pine, D. S., Smoller, J. W., Brown, C. H., …, Farber, G. K. (2016). Common measures for national institute of mental health funded research. Biological Psychiatry, 79, e91–e96. doi: 10.1016/j.biopsych.2015.07.006.
  2. Bauer, D. J., & Hussong, A. M. (2009). Psychometric approaches for developing commensurate measures across independent studies: Traditional and new models. Psychological Methods, 14, 101–125. doi: 10.1037/a0015583.CrossRefPubMedPubMedCentralGoogle Scholar
  3. Bauer, D. J., Howard, A. L., Baldasaro, R. E., Curran, P. J., Hussong, A. M., Chassin, L., & Zucker, R. A. (2013). A trifactor model for integrating ratings across multiple informants. Psychological Methods, 18, 475–493. doi: 10.1037/a0032475.CrossRefPubMedPubMedCentralGoogle Scholar
  4. Bollen, K., & Lennox, R. (1991). Conventional wisdom on measurement: a structural equation perspective. Psychological Bulletin, 110, 305–314.CrossRefGoogle Scholar
  5. Brown, C. H. (1983). Asymptotic comparison of missing data procedures for estimating factor loadings. Psychometrika, 48, 269–291.CrossRefGoogle Scholar
  6. Brown, C. H. (1990). Protecting against nonrandomly missing data in longitudinal studies. Biometrics, 46, 143–155.CrossRefPubMedGoogle Scholar
  7. Brown, C. H., Wyman, P. A., Brinales, J. M., & Gibbons, R. D. (2007). The role of randomized trials in testing interventions for the prevention of youth suicide. International Review of Psychiatry, 19, 617–631. doi: 10.1080/09540260701797779.CrossRefPubMedGoogle Scholar
  8. Brown, C. H., Wang, W., & Sandler, I. (2008). Examining how context changes intervention impact: the use of effect sizes in multilevel meta-analysis. Child Development Perspectives, 2, 198–205. doi: 10.1111/j.1750-8606.2008.00065.x.CrossRefPubMedPubMedCentralGoogle Scholar
  9. Brown, C. H., Sloboda, Z., Faggiano, F., Teasdale, B., Keller, F., Burhart, G., …, Prevention Science and Methodology Group (2013). Methods for synthesizing findings on moderation effects across multiple randomized trials. Prevention Science, 14, 144–156. doi: 10.1007/s11121-011-0207-8.CrossRefPubMedPubMedCentralGoogle Scholar
  10. Brown, C. H., Brincks, A., Huang, S., Perrino, T., Cruden, G., Pantin, H., …, Sandler, I. (2016). Two-year impact of prevention programs on adolescent depression: An integrative data analysis approach. Prevention Science, 1–21.Google Scholar
  11. Curran, P. J. (2009). The seemingly quixotic pursuit of a cumulative psychological science: Introduction to the special issue. Psychological Methods, 14, 77–80. doi: 10.1037/a0015972.CrossRefPubMedPubMedCentralGoogle Scholar
  12. Curran, P. J., & Hussong, A. M. (2009). Integrative data analysis: The simultaneous analysis of multiple data sets. Psychological Methods, 14, 81–100. doi: 10.1037/a0015914.CrossRefPubMedPubMedCentralGoogle Scholar
  13. Curran, P. J., Hussong, A. M., Cai, L., Huang, W., Chassin, L., Sher, K. J., & Zucker, R. A. (2008). Pooling data from multiple longitudinal studies: the role of item response theory in integrative data analysis. Developmental Psychology, 44, 365–380. doi: 10.1037/0012-1649.44.2.365.CrossRefPubMedPubMedCentralGoogle Scholar
  14. Dagne, G., Brown, C. H., Howe, G., Kellam, S., & Liu, L. (2016). Testing moderation in network meta-analysis with individual participant data. Statistics in Medicine, 35, 2485–2502. doi: 10.1002/sim.6883.CrossRefPubMedPubMedCentralGoogle Scholar
  15. Edwards, M. C., & Wirth, R. J. (2012). Valid measurement without factorial invariance: a longitudinal example. In J. R. Harring, G. R. Hancock, J. R. Harring, & G. R. Hancock (Eds.), Advances in longitudinal methods in the social and behavioral sciences (pp. 289–311). Charlotte: IAP Information Age Publishing.Google Scholar
  16. Gibbons, R. D., & Hedeker, D. R. (1992). Full-information item bi-factor analysis. Psychometrika, 57, 423–436.CrossRefGoogle Scholar
  17. Gibbons, R. D., Weiss, D. J., Pilkonis, P. A., Frank, E., Moore, T., Kim, J. B., & Kupfer, D. J. (2012). Development of a computerized adaptive test for depression. Archives of General Psychiatry, 69, 1104–1112. doi: 10.1001/archgenpsychiatry.2012.14.CrossRefPubMedPubMedCentralGoogle Scholar
  18. Hancock, G. R., Kuo, W.-L., & Lawrence, F. R. (2001). An illustration of second-order latent growth models. Structural Equation Modeling, 8, 470–489. doi: 10.1207/S15328007SEM0803_7.CrossRefGoogle Scholar
  19. Hedges, L. V., & Vevea, J. L. (1998). Fixed- and random-effects models in meta-analysis. Psychological Methods, 3, 486–504. doi: 10.1037/1082-989X.3.4.486.CrossRefGoogle Scholar
  20. Hendershot, T., Pan, H., Haines, J., Harlan, W. R., Marazita, M. L., McCarty, C. A., …, Hamilton, C. M. (2015). Using the PhenX Toolkit to add standard measures to a study. Current Protocols in Human Genetics, 86, 1.21.21–17. doi: 10.1002/0471142905.hg0121s86.
  21. Howe, G. W., Dagne, G., Brincks, A., & Beardslee, W. (2016). Evaluating construct equivalence and harmonizing measurement of adolescent depression when synthesizing results across multiple studies. Submitted for publicaton.Google Scholar
  22. Hussong, A. M., Curran, P. J., & Bauer, D. J. (2013). Integrative data analysis in clinical psychology research. Annual Review of Clinical Psychology, 9, 61–89. doi: 10.1146/annurev-clinpsy-050212-185522.CrossRefPubMedPubMedCentralGoogle Scholar
  23. Lavori, P. W., Brown, C. H., Duan, N., Gibbons, R. D., & Greenhouse, J. (2008). Missing data in longitudinal clinical trials. Part A. Design and conceptual issues. Psychiatric Annals, 38, 784–792.CrossRefPubMedPubMedCentralGoogle Scholar
  24. McArdle, J. J. (1988). Dynamic but structural equation modeling of repeated measures data. In J. R. Nesselroade & R. B. Cattell (Eds.), Handbook of multivariate experimental psychology (pp. 561–614). New York: Plenum Press.CrossRefGoogle Scholar
  25. McCarty, C. A., Huggins, W., Aiello, A. E., Bilder, R. M., Hariri, A., Jernigan, T. L., …, Zeng, Y. (2014). PhenX RISING: Real world implementation and sharing of PhenX measures. BMC Medical Genomics, 7, 16. doi: 10.1186/1755-8794-7-16.
  26. Muthén, B. O., & Brown, C. H. (2009). Estimating drug effects in the presence of placebo response: Causal inference using growth mixture modeling. Statistics in Medicine, 28, 3363–3395. doi: 10.1002/sim.3721.CrossRefPubMedPubMedCentralGoogle Scholar
  27. Muthén, B. O., Brown, C. H., Masyn, K., Jo, B., Khoo, S. T., Yang, C. C., …, Liao, J. (2002). General growth mixture modeling for randomized preventive interventions. Biostatistics, 3, 459–475. doi: 10.1093/biostatistics/3.4.459.
  28. Muthén, B. O., Brown, C. H., Leuchter, A., & Hunter, A. (2010). General approaches to analysis of course: Applying growth mixture modeling to randomized trials of depression medication. In P. E. Shrout, K. M. Keyes, & K. Ornstein (Eds.), Causality and psychopathology: finding the determinants of disorders and their cures (pp. 159–178). Washington: American Psychiatric Publishing.Google Scholar
  29. Pan, H., Tryka, K. A., Vreeman, D. J., Huggins, W., Phillips, M. J., Mehta, J. P., …, Ramos, E. M. (2012). Using PhenX measures to identify opportunities for cross-study analysis. Human Mutation, 33, 849–857. doi:  10.1002/humu.22074.
  30. Pathak, J., Pan, H., Wang, J., Kashyap, S., Schad, P. A., Hamilton, C. M., …, Chute, C. G. (2011). Evaluating phenotypic data elements for genetics and epidemiological research: Experiences from the eMERGE and PhenX Network Projects. AMIA Summits on Translational Science Proceedings, 2011, 41–45.Google Scholar
  31. Perrino, T., Howe, G., Sperling, A., Beardslee, W., Sandler, I., Shern, D., Brown, C. H.(2013). Advancing science through collaborative data sharing and synthesis. Perspectives on Psychological Science, 8(4), 433–444. doi: 10.1177/1745691613491579.
  32. Perrino, T., Beardslee, W., Bernal, G., Brincks, A., Cruden, G., Howe, G., …, Brown, C. H. (2015). Toward scientific equity for the prevention of depression and depressive symptoms in vulnerable youth. Prevention Science, 16, 642–651. doi: 10.1007/s11121-014-0518-7.
  33. Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: applications and data analysis methods. Thousand Oaks: Sage Publications.Google Scholar
  34. Royall, R. M., & Cumberland, W. G. (1981). An empirical study of the ratio estimator and estimators of its variance. Journal of the American Statistical Association, 76, 66–77.CrossRefGoogle Scholar
  35. Rubin, D. B. (1976). Inference and missing data. Biometrika, 63, 581–592.CrossRefGoogle Scholar
  36. Sandler, I. N., Wolchik, S. A., Cruden, G., Mahrer, N. E., Ahn, S., Brincks, A., & Brown, C. H. (2014). Overview of meta-analyses of the prevention of mental health, substance use, and conduct problems. Annual Review of Clinical Psychology, 10, 243–273. doi: 10.1146/annurev-clinpsy-050212-185524.CrossRefPubMedPubMedCentralGoogle Scholar
  37. Siddique, J., Brown, C. H., Hedeker, D., Duan, N., Gibbons, R. D., Miranda, J., & Lavori, P. W. (2008). Missing data in longitudinal trials—part B, analytic issues. Psychiatric Annals, 38, 793–801.CrossRefPubMedPubMedCentralGoogle Scholar
  38. Siddique, J., Crespi, C. M., Gibbons, R. D., & Green, B. L. (2011). Using latent variable modeling and multiple imputation to calibrate rater bias in diagnosis assessment. Statistics in Medicine, 30, 160–174. doi: 10.1002/sim.4109.CrossRefPubMedGoogle Scholar
  39. Siddique, J., Reiter, J. P., Brincks, A., Gibbons, R. D., Crespi, C. M., & Brown, C. H. (2015). Multiple imputation for harmonizing longitudinal non-commensurate measures in individual participant data meta-analysis. Statistics in Medicine, 34, 3399–3414. doi: 10.1002/sim.6562.CrossRefPubMedPubMedCentralGoogle Scholar
  40. Siddique, J., de Chavez, P. J., Howe, G., Cruden, G., & Brown, C. H. (2016). Limitations in using multiple imputation to harmonize individual participant data for meta-analysis. Prevention Science, 1–14.Google Scholar
  41. Tisak, J., & Meredith, W. (1990). Descriptive and associative developmental models. In A. Von Eye (Ed.), Statistical methods in longitudinal research (Vol. II, pp. 387–406). San Diego: Academic Press.Google Scholar
  42. Wang, C.-P., Brown, C. H., & Bandeen-Roche, K. (2005). Residual diagnostics for growth mixture models: Examining the impact of a preventive intervention on multiple trajectories of aggressive behavior. Journal of the American Statistical Association, 100, 1054–1076. doi: 10.1198/016214505000000501.CrossRefGoogle Scholar

Copyright information

© Society for Prevention Research 2017

Authors and Affiliations

  1. 1.Department of Epidemiology and Biostatistics, College of Human MedicineMichigan State UniversityEast LansingUSA
  2. 2.Department of Psychiatry and Behavioral SciencesFeinberg School of Medicine, Northwestern UniversityChicagoUSA
  3. 3.Department of Preventive MedicineFeinberg School of Medicine, Northwestern UniversityChicagoUSA
  4. 4.Department of PsychologyGeorge Washington UniversityWashingtonUSA
  5. 5.Department of BiostatisticsVanderbilt UniversityNashvilleUSA
  6. 6.School of Education and Human DevelopmentUniversity of MiamiMiamiUSA
  7. 7.Department of PsychologyArizona State UniversityTempeUSA
  8. 8.Department of Public Health Sciences, Miller School of MedicineUniversity of MiamiMiamiUSA

Personalised recommendations