Advertisement

Dealing with missing data by EM in single-case studies

  • Li-Ting ChenEmail author
  • Yanan Feng
  • Po-Ju Wu
  • Chao-Ying Joanne Peng
Article

Abstract

Single-case experimental design (SCED) research plays an important role in establishing and confirming evidence-based practices. Due to multiple measures of a target behavior in such studies, missing information is common in their data. The expectation–maximization (EM) algorithm has been successfully applied to deal with missing data in between-subjects designs, but only in a handful of SCED studies. The present study extends the findings from Smith, Borckardt, and Nash (2012) and Velicer and Colby (2005b, Study 2) by systematically examining the performance of EM in a baseline–intervention (or AB) design under various missing rates, autocorrelations, intervention phase lengths, and magnitudes of effects, as well as two fitted models. Three indicators of an intervention effect (baseline slope, level shift, and slope change) were estimated. The estimates’ relative bias, root-mean squared error, and relative bias of the estimated standard error were used to assess EM’s performance. The findings revealed that autocorrelation impacted the estimates’ qualities most profoundly. Autocorrelation interacted with missing rate in impacting the relative bias of the estimates, impacted the root-mean squared error nonlinearly, and interacted with the fitted model in impacting the relative bias of the estimated standard errors. A simpler model without autocorrelation can be used to estimate baseline slope and slope change in time-series data. EM is recommended as a principled method to handle missing data in SCED studies. Two decision trees are presented to assist researchers and practitioners in applying EM. Emerging research directions are identified for treating missing data in SCED studies.

Keywords

Missing data Single-case designs Autocorrelation Phase and slope change EM Effect size 

Notes

References

  1. Allison, D., & Gorman, B. (1993). Calculating effect sizes for meta-analysis: The case of the single case. Behavior Research and Therapy, 31, 621–631.  https://doi.org/10.1016/0005-7967(93)90115-B CrossRefGoogle Scholar
  2. Allison, D. B., Silverstein, J. M., & Gorman, B. S. (1996). Power, sample size estimation, and early stopping rules. In R. D. Franklin, D. B. Allison, & B. S. Gorman, (Eds.), Design and analysis of single-case research (pp. 335–371). Mahwah: Erlbaum.Google Scholar
  3. Barnard, J. (2000). Multiple imputation for missing data. Paper presented at the Summer Workshop of the Northeastern Illinois Chapter of the American Statistical Association, Northbrook.Google Scholar
  4. Box, G. E., & Jenkins, G. M. (1976). Time series analysis: Forecasting and control. San Francisco: Holden-Day.Google Scholar
  5. Chen, L.-T., Peng, C.-Y. J., & Chen, M.-E. (2015). Computing tools for implementing standards for single-case designs. Behavior Modification, 39, 835–869.  https://doi.org/10.1177/0145445515603706 CrossRefPubMedGoogle Scholar
  6. Cohen, J. (1969). Statistical power analysis for the behavioral sciences. New York: Academic Press.Google Scholar
  7. Couvreur, C. (1997). The EM algorithm: A guided tour. In M. Kárný & K. Warwick (Eds.), Computer intensive methods in control and signal processing (pp. 209–222). Boston: Birkhäuser.  https://doi.org/10.1007/978-1-4612-1996-5_12 CrossRefGoogle Scholar
  8. Crosbie, J. (1993). Interrupted time-series analysis with brief single-subject data. Journal of Consulting and Clinical Psychology, 61, 966–974.  https://doi.org/10.1037/0022-006X.61.6.966 CrossRefPubMedGoogle Scholar
  9. Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society: Series B, 39, 1–38.Google Scholar
  10. Dong, Y., & Peng, C.-Y. J. (2013). Principled missing data methods for researchers. SpringerPlus, 2, 222.  https://doi.org/10.1186/2193-1801-2-222
  11. Enders, C. K. (2010). Applied missing data analysis. New York: Guilford Press.Google Scholar
  12. Ferron, J. M., Farmer, J. L., & Owens, C. M. (2010). Estimating individual treatment effects from multiple-baseline data: A Monte Carlo study of multilevel-modeling approaches. Behavior Research Methods, 42, 930–943.  https://doi.org/10.3758/BRM.42.4.930 CrossRefPubMedGoogle Scholar
  13. Gustafson, S. A., Nassar, S. L., & Waddell, D. E. (2011). Single-case design in psychophysiological research. Part I: Context, structure, and techniques. Journal of Neurotherapy, 15, 18–34.  https://doi.org/10.1080/10874208.2011.545762 CrossRefGoogle Scholar
  14. Harrington, M, & Velicer, W. F. (2015). Comparing visual and statistical analysis in single-case studies using published studies. Multivariate Behavioral Research, 50, 162–183.  https://doi.org/10.1080/00273171.2014.973989 CrossRefPubMedPubMedCentralGoogle Scholar
  15. Hoogland, J. J., & Boomsma, A. (1998). Robustness studies in covariance structure modeling: An overview and a meta-analysis. Sociological Methods and Research, 26, 329–367.  https://doi.org/10.1177/0049124198026003003 CrossRefGoogle Scholar
  16. Horner, R. H., Carr, E. G., Halle, J., McGee, G., Odom, S., & Wolery, M. (2005). The use of single-subject research to identify evidence-based practice in special education. Exceptional Children, 71, 165–179.  https://doi.org/10.1177/001440290507100203 CrossRefGoogle Scholar
  17. Huitema, B. E., & McKean, J. W. (2000). Design specification issues in time-series intervention models. Educational and Psychological Measurement, 60, 38–58.  https://doi.org/10.1177/00131640021970358 CrossRefGoogle Scholar
  18. Huitema, B. E., & McKean, J. W. (2007). An improved portmanteau test for autocorrelated errors in interrupted time-series regression models. Behavior Research Methods, 39, 343–349.  https://doi.org/10.3758/BF03193002 CrossRefPubMedGoogle Scholar
  19. Kratochwill, T. R. (2015). Single-case research design and analysis: An overview. In T. R. Kratochwill & J. R. Levin (Eds.), Single-case research design and analysis: New directions for psychology and education (pp. 1–14). Mahwah: Erlbaum.Google Scholar
  20. Little, R. J. A., & Rubin, D. B. (1987). Statistical analysis with missing data. New York: Wiley.Google Scholar
  21. Little, R. J. A., & Rubin, D. B. (2002). Statistical analysis with missing data (2nd). New York: Wiley.CrossRefGoogle Scholar
  22. Manolov, R., & Solanas, A. (2008). Comparing N = 1 effect size indices in presence of autocorrelation. Behavior Modification, 32, 860–875.  https://doi.org/10.1177/0145445508318866 CrossRefPubMedGoogle Scholar
  23. Manolov, R., Solanas, A., Sierra, V., & Evans, J. J. (2011). Choosing among techniques for qualifying single-case intervention effectiveness. Behavior Therapy, 42, 533–545.  https://doi.org/10.1016/j.beth.2010.12.003 CrossRefPubMedGoogle Scholar
  24. Newman, D. A. (2014). Missing data: Five practical guidelines. Organizational Research Methods, 17, 372–411.  https://doi.org/10.1177/1094428114548590 CrossRefGoogle Scholar
  25. Parker, R. I., Brossart, D. R., Vannest, K. J., Long, J. R., De-Alba, R. G., Baugh, F. G., & Sullivan, J. R. (2005). Effect sizes in single case research: How large is large? School Psychology Review, 205, 116–132.Google Scholar
  26. Peng, C.-Y. J., Harwell, M., Liou, S.-M., & Ehman, L. H. (2006). Advances in missing data methods and implications for educational research. In S. Sawilowsky (Ed.), Real data analysis (pp. 31–78). Charlotte: Information Age.Google Scholar
  27. Richards, S. B., Taylor, R. L., & Ramasamy, R. (2014). Single subject research: Applications in educational and clinical settings (2nd). Belmont: Wadsworth.Google Scholar
  28. SAS Institute Inc. (2015). SAS/STAT 14.1 user’s guide. Cary: Author.Google Scholar
  29. Schafer, J. L. (1997). Analysis of incomplete multivariate data. London: Chapman & Hall/CRC.CrossRefGoogle Scholar
  30. Schafer, J. L., & Graham, J. W. (2002). Missing data: Our view of the state of the art. Psychological Methods, 7, 147–177.  https://doi.org/10.1037/1082-989X.7.2.147 CrossRefPubMedGoogle Scholar
  31. Schlomer, G. L., Bauman, S., & Card, N. (2010). Best practices for missing data management in counseling psychology. Journal of Counseling Psychology, 57, 1–10.  https://doi.org/10.1037/a0018082 CrossRefPubMedGoogle Scholar
  32. Shadish, W. R., Rindskopf, D. M., Hedges, L. V., & Sullivan, K. J. (2013). Bayesian estimates of autocorrelations in single-case designs. Behavior Research Methods, 43, 813–821.  https://doi.org/10.3758/s13428-012-0282-1 CrossRefGoogle Scholar
  33. Shadish, W. R., & Sullivan, K. J. (2011). Characteristics of single-case designs used to assess intervention effects in 2008. Behavior Research Methods, 43, 971–980.  https://doi.org/10.3758/s13428-011-0111-y CrossRefPubMedGoogle Scholar
  34. Smith, J. D. (2012). Single-case experimental designs: A systematic review of published research and current standards. Psychological Methods, 17, 510–550.  https://doi.org/10.1016/S0149-2063(99)80083-X CrossRefPubMedGoogle Scholar
  35. Smith, J. D., Borckardt, J. J., & Nash, M. R. (2012). Inferential precision in single-case time-series data streams: How well does the EM procedure perform when missing observations occur in autocorrelated data? Behavior Therapy, 43, 679–685.  https://doi.org/10.1016/j.beth.2011.10.001 CrossRefPubMedGoogle Scholar
  36. Solanas, A., Manolov, R., & Onghena, P. (2010). Estimating slope and level change in N = 1 designs. Behavior Modification, 34, 195–218.  https://doi.org/10.1177/0145445510363306 CrossRefPubMedGoogle Scholar
  37. Springer, T., & Urban, K. (2014). Comparison of the EM algorithm and alternatives. Numerical Algorithm 67(2), 335–364.  https://doi.org/10.1007/s11075-013-9794-8 CrossRefGoogle Scholar
  38. SPSS Inc. (2010). PASW Statistics for Windows and Mac (Version 18.0.0). Chicago: Author.Google Scholar
  39. Velicer, W. F., & Colby, S. M. (2005a). A comparison of missing-data procedures for ARIMA time-series analysis. Educational and Psychological Measurement, 65, 596–615.  https://doi.org/10.1177/0013164404272502 CrossRefGoogle Scholar
  40. Velicer, W. F., & Colby, S. M. (2005b). Missing data and general transformation approach to time series analysis. In A. Maydeu-Olivares & J. J. McArdle (Eds.), Contemporary psychometrics: A festschrift for Roderick P. McDonald (pp. 509–535). Mahwah: Erlbaum.Google Scholar
  41. Velicer, W. F., & Fava, J. (2003). Time series analysis. In J. Schinka & W. F. Velicer (Eds.), Handbook of psychology: Vol. 2. Research methods in psychology (pp. 581–606). New York: Wiley.Google Scholar
  42. What Works Clearinghouse. (2017). Standards handbook (Version 4.0). Retrieved from https://ies.ed.gov/ncee/wwc/Docs/referenceresources/wwc_standards_handbook_v4.pdf
  43. Wong, V. C., Wing, C., Steiner, P. M., Wong, M., & Cook, T. D. (2012). Research designs for program evaluation. In J. A. Schinka, & W. F. Velicer (Eds.), Handbook of psychology: Vol. 2. Research methods in psychology (2nd, pp. 316–341). Hoboken: Wiley.Google Scholar

Copyright information

© The Psychonomic Society, Inc. 2019

Authors and Affiliations

  • Li-Ting Chen
    • 1
    Email author
  • Yanan Feng
    • 2
  • Po-Ju Wu
    • 2
  • Chao-Ying Joanne Peng
    • 3
  1. 1.University of NevadaRenoUSA
  2. 2.Indiana UniversityBloomingtonUSA
  3. 3.National Taiwan UniversityTaipeiTaiwan

Personalised recommendations