Skip to main content

Advertisement

SpringerLink
Log in

Longitudinal Assessment Design and Statistical Power for Detecting an Intervention Impact

  • Published:
Prevention Science Aims and scope Submit manuscript

Abstract

In evaluating randomized control trials (RCTs), statistical power analyses are necessary to choose a sample size which strikes the balance between an insufficient and an excessive design, with the latter leading to misspent resources. With the growing popularity of using longitudinal data to evaluate RCTs, statistical power calculations have become more complex. Specifically, with repeated measures, the number and frequency of measurements per person additionally influence statistical power by determining the precision with which intra-individual change can be measured as well as the reliability with which inter-individual differences in change can be assessed. The application of growth mixture models has shown that the impact of universal interventions is often concentrated among a small group of individuals at the highest level of risk. General sample size calculations were consequently not sufficient to determine whether statistical power is adequate to detect the desired effect. Currently, little guidance exists to recommend a sufficient assessment design to evaluating intervention impact. To this end, Monte Carlo simulations are conducted to assess the statistical power and precision when manipulating study duration and assessment frequency. Estimates were extracted from a published evaluation of the proximal of the Good Behavior Game (GBG) on the developmental course of aggressive behavior. Results indicated that the number of time points and the frequency of assessments influence statistical power and precision. Recommendations for the assessment design of longitudinal studies are discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Notes

  1. It is acknowledged that there are several aspects which may influence the statistical power of a covariate effect in a GMM, e.g., number of profiles identified and degree of separation between profiles. For this study, undue emphasis is placed on the first two moments of the covariate impact on the rate of change as well as the reliability of the covariate estimate with respect to study duration and assessment frequency.

References

  • Adolph, K. E., Robinson, S. R., Young, J. W., & Gil-Alvarez, F. (2008). What is the shape of developmental change? Psychological Review, 3, 527–543.

    Article  Google Scholar 

  • Arbona, C. (2000). The development of academic achievement in school aged children: Precursors to career development. In S. D. Brown & R. W. Lent (Eds.), Handbook of Counseling Psychology (3rd ed., pp. 270–309). Hoboken: Wiley.

    Google Scholar 

  • Bloom, P. (2004). Myths of word learning. In D. G. Hall & S. R. Waxman (Eds.), Weaving a Lexicon (pp. 205–224). Cambridge: MIT Press.

    Google Scholar 

  • Bradshaw, C. P., Schaeffer, C. M., Petras, H., & Ialongo, N. (2010). Predicting negative life outcomes from early aggressive-disruptive behavior trajectories: Gender differences in maladaptation across life domains. Journal of Youth and Adolescence, 8, 953–966.

    Article  Google Scholar 

  • Broidy, L. M., Nagin, D. S., Tremblay, R. E., Bates, J. E., Brame, B., Dodge, K. A., Fergusson, D., Horwood, J. L., Loeber, R., Laird, R., Lyman, D. R., Moffitt, T. E., Pettit, G. S., & Vitaro, F. (2003). Developmental trajectories of childhood disruptive behaviors and adolescent delinquency: A six-site, cross-national study. Developmental Psychology, 39, 222–245.

    Article  PubMed  PubMed Central  Google Scholar 

  • Brown, C. H., & Liao, J. (1999). Principles for designing randomized preventive trials in mental health: An emerging developmental epidemiologic paradigm. American Journal of Community Psychology, 27, 677–714.

    Article  Google Scholar 

  • Brown, C. H., Wang, W., Kellam, S. G., Muthén, B. O., Petras, H., Toyinbo, P., Poduska, J., Ialongo, N., Wyman, P. A., Chamberlain, P., Sloboda, Z., MacKinnon, D. P., & Windham, A. (2008). Methods for testing theory and evaluating impact in randomized field trials: Intent-to-treat analyses for integrating the perspectives of person, place, and time. Drug and Alcohol Dependence, 95, S74–S104.

    Article  PubMed  PubMed Central  Google Scholar 

  • Catalano, R. F., Kosterman, R., Hawkins, J. D., Newcomb, M. D., & Abbott, R. D. (1996). Modeling the etiology of adolescent substance use: A test of the social development model. Journal of Drug Issues, 26, 429–455.

    Article  PubMed  PubMed Central  Google Scholar 

  • Cohen, J. (1969). Statistical power analysis for the behavioral sciences. Hillsdale: Lawrence Erlbaum Associates.

    Google Scholar 

  • Collins, L. M. (2006). Analysis of longitudinal data: The integration of theoretical model, temporal design, and statistical model. Annual Review of Psychology, 57, 505–528.

    Article  PubMed  Google Scholar 

  • Collins, L. M., & Graham, J. W. (2002). The effect of the timing and spacing of observations in longitudinal studies of tobacco and other drug use: Temporal design considerations. Drug and Alcohol Dependence, 68, S85–S96.

    Article  PubMed  Google Scholar 

  • Cook, N. R., & Ware, J. H. (1983). Design and analysis for longitudinal research. Annual Review of Public Health, 4, 1–23.

    Article  CAS  PubMed  Google Scholar 

  • Fitzmaurice, G. M., Laird, N. M., & Ware, J. H. (2004). Applied longitudinal data analysis. Hoboken, NJ: Wiley.

  • Griffin, K. W., Botvin, G. J., & Nichols, T. R. (2006). Effects of a school-based drug abuse prevention program for adolescents on HIV risk behaviors in young adulthood. Prevention Science, 7, 103–112.

    Article  PubMed  Google Scholar 

  • Hedeker, D., Gibbons, R. D., & Waternaux, C. (1999). Sample size estimation for longitudinal designs with attrition: Comparing time-related contrasts between two groups. Journal of Educational and Behavioral Statistics, 24, 70–93.

    Article  Google Scholar 

  • Kellam, S. G., Brown, C. H., Poduska, J. M., Ialongo, N. S., Wang, W., Toyinbo, P., Petras, H., Ford, C., Windham, A., & Wilcox, H. C. (2008). Effects of a universal classroom behavior management program in first and second grades on young adult behavioral, psychiatric, and social outcomes. Drug and Alcohol Dependence, 95, S5–S28.

    Article  PubMed  PubMed Central  Google Scholar 

  • Kellam, S. G., Koretz, D., & Mościcki, E. K. (1999). Core elements of developmental epidemiologically based prevention research. American Journal of Community Psychology, 27(4), 463–82.

    Article  CAS  PubMed  Google Scholar 

  • Kellam, S. G., Mackenzie, A. C., Brown, C. H., Poduska, J. M., Wang, W., Petras, H., & Wilcox, H. C. (2011). The good behavior game and the future of prevention and treatment. Addiction Science and Clinical Practice, 6(1), 73–84.

    PubMed  PubMed Central  Google Scholar 

  • Kellam, S., & Rebok, G. (1992). Building developmental and etiological theory through epidemiological based preventive intervention trials (pp. 162–195). In J. McCord & R. E. Tremblay (Eds.), Preventing antisocial behavior: Interventions from birth through adolescence. New York: Neale Watson Academic Publishers.

    Google Scholar 

  • Lipsey, M. W., & Hurley, S. M. (2009). Design sensitivity: Statistical power for applied experimental research. In L. Bickman & D. J. Rog (Eds.), The SAGE Handbook of Applied Social Research Methods (pp. 44–76). California: SAGE Publications, Inc.

    Chapter  Google Scholar 

  • Loeber, R., & Farrington, D. P. (2000). Young children who commit crime: Epidemiology, developmental origins, risk factors, early interventions, and policy implications. Development and Psychopathology, 12, 737–762.

    Article  CAS  PubMed  Google Scholar 

  • Mason, W. A., Kosterman, R., Hawkins, J. D., Haggerty, K. P., & Spoth, R. (2003). Reducing adolescents’ growth in substance use and delinquency: Randomized trial effects of a preventive parent training intervention. Prevention Science, 4, 203–2013.

  • Macleod, J., Oakes, R., Copello, A., Crome, I., Egger, M., Hickman, M., Oppenkowski, T., Stokes-Lampard, H., & Davey, S. G. (2004). Psychological and social sequelae of cannabis and other illicit drug use by young people: A systematic review of longitudinal, general population studies. Lancet, 363(9421), 1579–88.

    Article  PubMed  Google Scholar 

  • Maxwell, S. E., Kelley, K., & Rausch, J. R. (2008). Sample size planning for statistical power and accuracy in parameter estimation. Annual Review of Psychology, 59, 537–63.

    Article  PubMed  Google Scholar 

  • McGrath, J. E., & Tschan, F. (2004). Temporal matters in social psychology: Examining the role of time in the lives of groups and individuals. Washington, DC: APA Publications.

    Book  Google Scholar 

  • Moerbeek, M. (2008). Powerful and cost-efficient designs for longitudinal intervention studies with two treatment groups. Journal of Educational and Behavioral Statistics, 33, 41–61.

    Article  Google Scholar 

  • Moffitt, T. E., & Caspi, A. (2001). Childhood predictors differentiate life-course-persistent and adolescence-limited antisocial pathways among males and females. Development and Psychopathology, 13(2), 355–375.

    Article  CAS  PubMed  Google Scholar 

  • Mooney, C. Z. (1997). Monte Carlo simulation. Thousand Oaks: Sage Publications.

    Book  Google Scholar 

  • Muthén, B. O., & Curran, P. J. (1997). General longitudinal modeling of individual differences in experimental design: A latent variable framework for analysis and power estimation. Psychological Methods, 4(2), 371–402.

    Article  Google Scholar 

  • Muthén, B., & Shedden, K. (1999). Finite mixture modeling with mixture outcomes using the EM algorithm. Biometrics, 6, 463–469.

    Article  Google Scholar 

  • Muthén, B. (2002). Using Mplus Monte Carlo simulations in practice: A note on assessing estimation quality and power in latent variable models. Version 2, March 22, 2002. Retrieved from www.statmodel.com on 05/15/2015.

  • Muthén, B. (2004). Latent variable analysis: Growth mixture modeling and related techniques for longitudinal data. In D. Kaplan (Ed.), Handbook of quantitative methodology for the social sciences (pp. 345–368). Newbury Park: Sage Publications.

    Google Scholar 

  • Muthén, B. and Asparouhov, T. (2002). Using Mplus Monte Carlo simulations in practice: A note on non-normal missing data in latent variable models. Version 2, March 22, 2002. Retrieved from www.statmodel.com on 05/15/2015.

  • Muthén, B., Brown, C. H., Masyn, K., Jo, B., Khoo, S. T., Yang, C. C., Wang, C. P., Kellam, S., Carlin, J., & Liao, J. (2002). General growth mixture modeling for randomized preventive interventions. Biostatistics, 3, 459–475.

    Article  PubMed  Google Scholar 

  • Muthén, L.K. & Muthén, B. (1998-2014). Mplus User’s Guide. Seventh Edition. Los Angeles, CA: Muthén & Muthén.

  • Muthén, B., Brown, C. H., Hunter, A., Cook, I. A., & Leuchter, A. F. (2011). General approaches to analysis of course: Applying growth mixture modeling to randomized trials of depression medication. In P. E. Shrout (Ed.), Causality and Psychopathology: Finding the Determinants of Disorders and their Cures (pp. 159–178). New York: Oxford University Press.

    Google Scholar 

  • Nagin, D. S. (1999). Analyzing developmental trajectories: A semiparametric, group-based approach. Psychological Methods, 4, 139–157.

    Article  Google Scholar 

  • Nagin, D., & Tremblay, R. E. (2001). Parental and early childhood predictors of persistent physical aggression in boys from kindergarten to high school. Archives of General Psychiatry, 58, 389–394.

    Article  CAS  PubMed  Google Scholar 

  • Patterson, G. R., DeBaryshe, B. D., & Ramsey, E. (1989). A developmental perspective on antisocial behavior. American Psychologist, 44, 329–335.

    Article  CAS  PubMed  Google Scholar 

  • Paxton, P., Curran, P. J., Bollen, K. A., Kirby, J., & Chen, F. (2001). Monte Carlo experiments: Design and implementation. Structural Equation Modeling, 8, 287–312.

    Article  Google Scholar 

  • Petras, H., Kellam, S. G., Brown, H., Muthén, B., Ialongo, N., & Poduska, J. M. (2008). Developmental epidemiological courses leading to antisocial personality disorder and violent and criminal behavior: Effects by young adulthood of a universal preventive intervention in first- and second-grade classrooms. Drug and Alcohol Dependence, 95, S45–S59.

    Article  PubMed  PubMed Central  Google Scholar 

  • Petras, H., Masyn, K., & Ialongo, N. (2011). The distal impact of two first grade preventive interventions on aggressive/disruptive behavior in adolescence: An application of latent transition growth mixture modeling. Prevention Science, 12, 300–313.

    Article  PubMed  PubMed Central  Google Scholar 

  • Piquero, A. R. (2008). Taking stock of developmental trajectories of criminal activity over the life course. In A. M. Lieberman (Ed.), The long view of crime: A synthesis of longitudinal research (pp. 23–78). New York: Springer.

    Chapter  Google Scholar 

  • Raudenbush, S. W. (1993). Hierarchical linear models and experimental design. In L. K. Edwards (Ed.), Applied analysis of variance in behavioral science. New York: Marcel Dekker.

    Google Scholar 

  • Raudenbush, S. W., & Xiao-Feng, L. (2001). Effects of study duration, frequency of observation, and sample size on power in studies of group differences in polynomial change. Psychological Methods, 6(4), 387–401.

    Article  CAS  PubMed  Google Scholar 

  • Schlesselman, J. J. (1973). Planning a longitudinal study: II. Frequency of measurement and study duration. Journal of Chronic Disease, 26, 561–570.

    Article  CAS  Google Scholar 

  • Schmiedek, F., Lövdén, M., & Lindenberger, U. (2009). On the relation of mean reaction time and intraindividual reaction time variability. Psychology and Aging, 24, 841–857.

    Article  PubMed  Google Scholar 

  • Singer, J. D., & Willet, J. B. (1996). Methodological issues in the design of longitudinal research: Principles and recommendations for a quantitative study of teacher’s careers. Educational Evaluation and Policy Analysis, 4, 265–283.

    Article  Google Scholar 

  • van Dulmen, M. H. M., Goncy, E. A., Vest, A., & Flannery, D. J. (2009). Group-based trajectory modeling of externalizing behavior problems from childhood through adulthood: Exploring discrepancies in the empirical findings. In J. Savage (Ed.), The development of persistent criminality (pp. 288–314). New York: Oxford University.

    Chapter  Google Scholar 

  • Werthamer-Larsson, L., Kellam, S. G., & Wheeler, L. (1991). Effect of first-grade classroom environment on child shy behavior, aggressive behavior, and concentration problems. American Journal of Community Psychology, 19, 585–602.

    Article  CAS  PubMed  Google Scholar 

  • Willett, J. B. (1989). Some results on reliability for the longitudinal measurement of change: Implications for the decision of studies of individual growth. Educational and Psychological Measurement, 49, 587–601.

    Article  Google Scholar 

  • Wilson, J. W., & Lipsey, M. W. (2007). School-based interventions for aggressive and disruptive behavior: Update of a meta-analysis. American Journal of Preventive Medicine, 33(2 Suppl.), S130–S143.

    Article  PubMed  PubMed Central  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. American Institutes for Research, 1000 Thomas Jefferson Street, NW, Washington, DC, 20007-3835, USA

    Hanno Petras

Authors
  1. Hanno Petras
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Hanno Petras.

Ethics declarations

Conflict of Interest

The author declares that he has no conflict of interest.

Informed Consent

Informed consent was obtained from all individual participants included in the study.

Ethical Approval

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.

Additional information

Earlier versions of this manuscript were presented at an expert panel meeting on “Impact of Early Interventions on Trajectories of Violence” in October of 2010 organized by CDC as well as at the Society for Prevention Research meeting in 2011. The helpful feedback from Drs. Brincks (University of Miami), Greenbaum (University of South Florida), and Capaldi (Oregon Social Learning Center) as well as from the Prevention Science Methodology Group (PSMG) are gratefully acknowledged.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Petras, H. Longitudinal Assessment Design and Statistical Power for Detecting an Intervention Impact. Prev Sci 17, 819–829 (2016). https://doi.org/10.1007/s11121-016-0646-3

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11121-016-0646-3

Keywords

Search

Navigation