Abstract
Selecting a quantitative measure to guide decision making in single-case experimental designs (SCEDs) is complicated. Many measures exist and all have been rightly criticized. The two general classes of measure are overlap-based (e.g., percentage nonoverlapping data) and distance-based (e.g., Cohen’s d). We compare several measures from each category for Type I error rate and power across a range of designs using equal numbers of observations (i.e., 3–10) in each phase. Results showed that Tau and the distance-based measures (i.e., RD and g) provided the highest decision accuracies. Other overlap-based measures (e.g., PND, dual-criterion method) did not perform as well. It is recommended that Tau be used to guide decision making about the presence/absence of a treatment effect, and RD or g be used to quantify the magnitude of the treatment effect.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Allison, D. B., & Gorman, B. S. (1993). Calculating effect sizes for meta-analysis: The case of the single case. Behaviour Research & Therapy, 31, 621–631. https://doi.org/10.1016/0005-7967(93)90115-B
Bar-Hillel, M., & Wagenaar, W. A. (1991). The perception of randomness. Advances in Applied Mathematics, 12, 428–454. https://doi.org/10.1016/0196-8858(91)90029-I
Branch, M. N. (1999). Statistical inference in behavior analysis: Some things significance testing does and does not do. The Behavior Analyst, 22, 87–92. https://doi.org/10.1007/BF03391984
Branch, M. (2014). Malignant side effects of null-hypothesis significance testing. Theory & Psychology, 24, 256–277. https://doi.org/10.1177/0959354314525282
Carlin, M. T., & Costello, M. S. (2018). Development of a distance-based effect size metric for single-case research: Ratio of distances. Behavior Therapy, 49, 981–994. https://doi.org/10.1016/j.beth.2018.02.005
Carter, M. (2013). Reconsidering overlap-based measures for quantitative synthesis of single-subject data: What they tell us and what they don’t. Behavior Modification, 37, 378–390. https://doi.org/10.1177/0145445513476609
Cohen, J. (1988). Statistical power analysis for the behavioral sciences ((2nd ed.). ed.).
DeProspero, A., & Cohen, S. (1979). Inconsistent visual analysis of intrasubject data. Journal of Applied Behavior Analysis, 12, 573–579. https://doi.org/10.1901/jaba.1979.12-573
Fisher, W. W., Kelley, M. E., & Lomas, J. E. (2003). Visual aids and structured criteria for improving visual inspection and interpretation of single-case designs. Journal of Applied Behavior Analysis, 36, 387–406. https://doi.org/10.1901/jaba.2003.36-387
Hahn, U., & Warren, P. A. (2009). Perceptions of randomness: Why three heads is better than four. Psychological Review, 116, 454–461. https://doi.org/10.1037/a0015241
Lakens, D. (2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests ad ANOVAs. Frontiers in Psychology, 4, Article 863, 1–12. https://doi.org/10.3389/fpsyg.2013.00863
Lanovaz, M. J., Giannakakos, A. R., & Destras, O. (2020). Machine learning to analyze single-case data: A proof of concept. Perspectives on Behavior Science, 43, 21–38. https://doi.org/10.1007/s40614-020-00244-0
Ma, H. (2006). An alternative method for quantitative synthesis of single-subject researches: Percentage of data points exceeding the median. Behavior Modification, 30, 598–617. https://doi.org/10.1177/0145445504272974
Manolov, R., & Solanas, A. (2008). Comparing n = 1 effect size indices in presence of autocorrelation. Behavior Modification, 32, 860–875. https://doi.org/10.1177/0145445508318866
McKnight, S. D., McKean, J. W., & Huitema, B. E. (2000). A double bootstrap method to analyze linear models with autoregressive error terms. Psychological Methods, 5, 87–101. https://doi.org/10.1037/1082-989X.5.1.87
Nickerson, R. (2002). The production and perception of randomness. Psychological Review, 109, 330–357. https://doi.org/10.1037/0033-295X.109.2.330
Ninci, J., Vannest, K. J., Wilson, V., & Zhang, N. (2015). Interrater agreement between visual analysts of single-case data: A meta-analysis. Behavior Modification, 39, 510–541. https://doi.org/10.1177/0145445515581327
Parker, R. I., Vannest, K. J., Davis, J. L., & Sauber, S. B. (2011). Combining non-overlap and trend for single-case research: Tau-U. Behavior Therapy, 42, 284–299. https://doi.org/10.1016/j.beth.2010.08.006
Pustejovsky, J. E. (2019). Procedural sensitivities of effect sizes for single-case designs with directly observed behavioral outcome measures. Psychological Methods, 24, 217–235. https://doi.org/10.1037/met0000179
Scruggs, T. E., & Mastropieri, M. A. (1998). Summarizing single-subject research: Issues and applications. Behavior Modification, 22, 221–242. https://doi.org/10.1177/01454455980223001
Scruggs, T. E., & Mastropieri, M. A. (2013). PND at 25: Past, present, and future trends in summarizing single-subject research. Remedial & Special Education, 34, 9–19. https://doi.org/10.1177/0741932512440730
Scruggs, T. E., Mastropieri, M. A., & Casto, G. (1987). The quantitative synthesis of single-case research: Methodology and validation. Remedial & Special Education, 8, 24–33. https://doi.org/10.1177/074193258700800206
Skinner, B. F. (1938). The behavior of organisms: An experimental analysis. Appleton-Century.
Voss, J. L., Federmeier, K. D., & Paller, K. A. (2012). The potato chip really does look like Elvis! Neural hallmarks of conceptual processing associated with finding novel shapes subjectively meaningful. Cerebral Cortex, 22, 2354–2364. https://doi.org/10.1093/cercor/bhr315
Wolery, M., Busick, M., Reichow, B., & Barton, E. E. (2010). Comparison of overlap methods for quantitatively synthesizing single-subject data. Journal of Special Education, 44, 18–28. https://doi.org/10.1177/0022466908328009
Wolfe, K., Seaman, M. A., & Drasgow, E. (2016). Interrater agreement on the visual analysis of individual tiers and functional relations in multiple baseline designs. Behavior Modification, 40, 852–873. https://doi.org/10.1177/0145445516644699
Wolfe, K., Seaman, M. A., Drasgow, E., & Sherlock, P. (2018). An evaluation of the agreement between the conservative dual-criterion method and expert visual analysis. Journal of Applied Behavior Analysis, 51, 345–351. https://doi.org/10.1002/jaba.453
Availability of Data
Upon request
Code Availability
Not applicable
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflicts of Interest
None
Ethics Approval
Not applicable
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Carlin, M.T., Costello, M.S. Statistical Decision-Making Accuracies for Some Overlap- and Distance-based Measures for Single-Case Experimental Designs. Perspect Behav Sci 45, 187–207 (2022). https://doi.org/10.1007/s40614-021-00317-8
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40614-021-00317-8