Type I Error Rates from Mixed Effects Model Repeated Measures Versus Fixed Effects Anova with Missing Values Imputed Via Last Observation Carried Forward

Abstract

Treatment effects are often evaluated by comparing change over time in outcome measures. However, valid analyses of longitudinal data can be problematic when subjects discontinue (dropout) prior to completing the trial. This study compared the Type I error rates from a likelihood-based repeated measures analysis (MMRM) to a fixed-effects analysis of variance where missing values were imputed using the last observation carried forward approach (LOCF). Comparisons were made in 32 scenarios, with 3000 simulated data sets per scenario. The null hypothesis of no difference between treatments in mean change from baseline to endpoint was true in all data sets. Subject dropout was introduced to generate ignorable and nonignorable missingness.

Pooled across all scenarios, the Type I error rates for MMRM and LOCF were 5.85% and 10.36%, respectively. Type I error rates in the 32 scenarios ranged from 5.03% to 7.17% for MMRM, and from 4.43% to 36.30% for LOCF. In 19 of the 32 scenarios, MMRM yielded a Type I error rate that was at least 1.00% closer to the expected rate of 5.00% than the corresponding rate from LOCF.

Greater inflation of Type I error in LOCF resulted from greater bias in estimates of mean change from baseline to endpoint and unduly small standard errors that were a consequence of failing to account for the uncertainty of imputation. The superior control of Type I error by MMRM suggested that MMRM should replace LOCF as the default primary analysis for longitudinal clinical trials where dropout bias may exist.

This is a preview of subscription content, access via your institution.

References

  1. 1.

    Milliken GA, Johnson DE. “The Analysis of Messy Data.” In Vol. I. Designed Experiments. New York, NY: Chapman & Hall; 1993:326.

    Google Scholar 

  2. 2.

    Gibbons RD, Hedeker D, Elkin I, Waternaux C, Kraemer HC, Greenhouse JB, Shea T, Imber S, Stotsky SM, Watkins JT. Some conceptual and statistical issues in analysis of longitudinal psychiatric data. Arch Gen Psych. 1993;50:739–750.

    CAS  Article  Google Scholar 

  3. 3.

    Laird NM. Missing data in longitudinal studies. Stat Med. 1988;7:305–315.

    CAS  Article  Google Scholar 

  4. 4.

    Little R, Rubin D. Statistical Analysis with Missing Data. New York, NY: John Wiley and Sons: 1987.

    Google Scholar 

  5. 5.

    Lavori PW. Clinical trials in psychiatry: Should protocol deviation censor patient data? Neuropsycho-pharmacol. 1992;6(1):39–48.

    Google Scholar 

  6. 6.

    Verbeke G, Molenberghs G. Linear Mixed Models for Longitudinal Data. New York, NY: Springer; 2000.

    Google Scholar 

  7. 7.

    Littell RC, Milliken GA, Stroup WW, Wolfinger RD. The SAS System for Mixed Models. Cary, NC: SAS Institute Inc; 1996; Chap. 1–10.

    Google Scholar 

  8. 8.

    Cnaan A, Laird NM, Slasor P. Using the general linear mixed model to analyze unbalanced repeated measures and longitudinal data. Stat Med. 1997; 16: 2349–2380.

    CAS  Article  Google Scholar 

  9. 9.

    Lavori PW, Dawson R, Shera D. A multiple imputation strategy for clinical trials with truncation of patient data. Stat Med. 1995;14:1913–1925.

    CAS  Article  Google Scholar 

  10. 10.

    Siddiqui O, Ali MW. A comparison of the random-effects pattern mixture model with last-observation-carried-forward (LOCF) analysis in longitudinal clinical trials with dropouts. J Biopharm Stat. 1998;8(4):545–563.

    CAS  Article  Google Scholar 

  11. 11.

    Heyting A, Tolboom JTBM, Essers JGA. Statistical handling of dropouts in longitudinal clinical trials. Stat Med. 1992;11:2043–2061.

    CAS  Article  Google Scholar 

  12. 12.

    Mallinckrodt CH, Clark WS, David SR. Accounting for dropout bias using mixed-effects models. J Bio-Pharm Stat. Forthcoming.

  13. 13.

    Kay SR, Opler LA, Fiszbein A. Positive and Negative Syndrome Scale (PANSS) Manual. North Tonawanda, NY: Multi-Health Systems; 1986.

    Google Scholar 

  14. 14.

    Tollefson GD, Beasley CM, Jr., Tran P, Street JS, Krueger JA, Tamura RN, Graffeo KA, Thieme ME. Olanzapine versus Haloperidol in the treatment of schizophrenia, schizoaffective, and schizophreniform disorders: Results of an international collaborative trial. Am J Psychiatry. 1997;154(4):457–465.

    CAS  Article  Google Scholar 

  15. 15.

    Van Vleck LD, Gregory KE. Multiple trait restricted maximum likelihood for simulated measures of ovulation rate with underlying multivariate normal distributions. J Anim Sci. 1992;70:57.

    Article  Google Scholar 

  16. 16.

    Mallinckrodt CH. The Effects of Animal Model Approximations and Data Problems on the Reliability of Genetic Evaluations. Ph.D. Dissertation. Fort Collins, CO: Colorado State University; 1993.

    Google Scholar 

  17. 17.

    Littell RC, Milliken GA, Stroup WW, Wolfinger RD. The SAS System for Mixed Models. Cary, NC: SAS Institute Inc; 1996;499.

    Google Scholar 

  18. 18.

    Littell RC, Milliken GA, Stroup WW, Wolfinger RD. The SAS System for Mixed Models. Cary, NC: SAS Institute Inc; 1996;38.

    Google Scholar 

  19. 19.

    Rubin D, Shenker N. Multiple imputations in health care databases: An overview and some applications. Stat Med. 1991;10:585–598.

    CAS  Article  Google Scholar 

  20. 20.

    Little R, Yau L. Intent-to-treat analysis for longitudinal studies with drop-outs. Biometrics. 1996;52: 1324–1333.

    CAS  Article  Google Scholar 

Download references

Author information

Affiliations

Authors

Corresponding author

Correspondence to Craig H. Mallinckrodt.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Mallinckrodt, C.H., Clark, W.S. & David, S.R. Type I Error Rates from Mixed Effects Model Repeated Measures Versus Fixed Effects Anova with Missing Values Imputed Via Last Observation Carried Forward. Ther Innov Regul Sci 35, 1215–1225 (2001). https://doi.org/10.1177/009286150103500418

Download citation

Key Words

  • Missing data
  • Mixed-effects models
  • Dropout bias
  • LOCF