Abstract
Group-based experimental designs are an outgrowth of the logic of null-hypothesis significance testing and thus, statistical tests are often considered inappropriate for single-case experimental designs. Behavior analysts have recently been more supportive of efforts to include appropriate statistical analysis techniques to evaluate single-case experimental design data. One way that behavior analysts can incorporate statistical analyses into their practices with single-case experimental designs is to use Monte Carlo analyses. These analyses compare experimentally obtained behavioral data to simulated samples of behavioral data to determine the likelihood that the experimentally obtained results occurred due to chance (i.e., a p value). Monte Carlo analyses are more in line with behavior analytic principles than traditional null-hypothesis significance testing. We present an open-source Monte Carlo tool, created in shiny, for behavior analysts who want to use Monte Carlo analyses in addition as part of their data analysis.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
For simplicity, we are omitting a discussion of nonparametric tests that do not have the same assumptions as parametric tests and regression techniques that purport to describe whether an independent variable affects the trajectory of the dependent variable.
The Microsoft support page for saving an Excel workbook as a CSV file can be found here: https://support.microsoft.com/en-us/office/save-a-workbook-to-text-format-txt-or-csv-3e9a9d6c-70da-4255-aa28-fcacf1f081e6.
References
Ator, N. A. (1999). Statistical inference in behavior analysis: Environmental determinants? The Behavior Analyst, 22(2), 93–97. https://doi.org/10.1007/BF03391985
Baggio, A., & Langendoen, K. (2008). Monte Carlo localization for mobile wireless sensor networks. Ad Hoc Networks, 6(5), 718–733. https://doi.org/10.1016/j.adhoc.2007.06.004
Berry, M. S., Sweeney, M. M., & Odum, A. L. (2014). Effects of baseline reinforcement rate on operant ABA and ABC renewal. Behavioural Processes, 108, 87–93. https://doi.org/10.1016/j.beproc.2014.09.009
Branch, M. N. (1999). Statistical inference in behavior analysis: Some things significance testing does and does not do. The Behavior Analyst, 22(2), 87–92. https://doi.org/10.1007/BF03391984
Branch, M. (2014). Malignant side effects of null-hypothesis significance testing. Theory & Psychology, 24(2), 256–277. https://doi.org/10.1177/0959354314525282
Branch, M. N. (2019). The "reproducibility crisis:" Might the methods used frequently in behavior-analysis research help? Perspectives on Behavior Science, 42(1), 77–89. https://doi.org/10.1007/s40614-018-0158-5
Chang, W., Cheng, J., Allaire, J. J., Xie, Y., & McPherson, J. (2020). shiny: Web application framework for R. R package version 1.5.0. https://CRAN.R-project.org/package=shiny
Cooper, J. O., Heron, T. E., & Heward, W. L. (2020). Applied behavior analysis (3rd ed.). Pearson Education.
Craig, A. R., & Fisher, W. W. (2019). Randomization tests as alternative analysis methods for behavior-analytic data. Journal of the Experimental Analysis of Behavior, 111(2), 309–328. https://doi.org/10.1002/jeab.500
Crosbie, J. (1999). Statistical inference in behavior analysis: Useful friend. The Behavior Analyst, 22(2), 105–108. https://doi.org/10.1007/BF03391987
DeHart, W. B., & Kaplan, B. A. (2019). Applying mixed-effects modeling to single-subject designs: An introduction. Journal of the Experimental Analysis of Behavior, 111(2), 192–206. https://doi.org/10.1002/jeab.507
Ferron, J. M., Farmer, J. L., & Owens, C. M. (2010). Estimating individual treatment effects from multiple-baseline data: A Monte Carlo study of multilevel-modeling approaches. Behavior Research Methods, 42(4), 930–943. https://doi.org/10.3758/BRM.42.4.930
Ferron, J. M., Joo, S. H., & Levin, J. R. (2017). A Monte Carlo evaluation of masked visual analysis in response-guided versus fixed-criteria multiple-baseline designs. Journal of Applied Behavior Analysis, 50(4), 701–716. https://doi.org/10.1002/jaba.410
Field, A. (2018). Discovering statistics using IBM SPSS statistics (5th ed.). Sage Publications.
Fisher, R. A. (1935). The design of experiments. Oliver & Boyd.
Friedel, J. E., DeHart, W. B., & Odum, A. L. (2017). The effects of 100 dB 1-kHz and 22-kHz tones as punishers on lever pressing in rats. Journal of the Experimental Analysis of Behavior, 107(3), 354–368. https://doi.org/10.1002/jeab.254
Friedel, J. E., DeHart, W. B., Foreman, A. M., & Andrew, M. E. (2019a). A Monte Carlo method for comparing generalized estimating equations to conventional statistical techniques for discounting data. Journal of the Experimental Analysis of Behavior, 111(2), 207–224. https://doi.org/10.1002/jeab.497
Friedel, J. E., Galizio, A., Berry, M. S., Sweeney, M. M., & Odum, A. L. (2019b). An alternative approach to relapse analysis: Using Monte Carlo methods and proportional rates of response. Journal of the Experimental Analysis of Behavior, 111(2), 289–308. https://doi.org/10.1002/jeab.489
Galizio, A., Frye, C. C. J., Haynes, J. M., Friedel, J. E., Smith, B. M., & Odum, A. L. (2018). Persistence and relapse of reinforced behavioral variability. Journal of the Experimental Analysis of Behavior, 109(1), 210–237. https://doi.org/10.1002/jeab.309
Giannakakos, A. R., & Lanovaz, M. J. (2019). Using AB designs with nonoverlap effect size measures to support clinical decision-making: A Monte Carlo validation. Behavior Modification, 1–16. Advance online publication. https://doi.org/10.1177/0145445519860219
Gilroy, S. P., & Hantula, D. A. (2018). Discounting model selection with area-based measures: A case for numerical integration. Journal of the Experimental Analysis of Behavior, 109(2), 433–449. https://doi.org/10.1002/jeab.318
Haddock, J. N., & Iwata, B. A. (in press). Software for graphing time-series data. Journal of Applied Behavior Analysis.
Hales, A. H., Wesselmann, E. D., & Hilgard, J. (2019). Improving psychological science through transparency and openness: An overview. Perspectives on Behavior Science, 42(1), 13–31. https://doi.org/10.1007/s40614-018-00186-8
Horner, R. D., & Baer, D. M. (1978). Multiple-probe technique: A variation on the multiple baseline. Journal of Applied Behavior Analysis, 11(1), 189–196. https://doi.org/10.1901/jaba.1978.11-189
Jacobs, K. W. (2019). Replicability and randomization test logic in behavior analysis. Journal of the Experimental Analysis of Behavior, 111(2), 329–341. https://doi.org/10.1002/jeab.501
Killeen, P. R. (2019). Predict, control, and replicate to understand: How statistics can foster the fundamental goals of science. Perspectives on Behavavior Science, 42(1), 109–132. https://doi.org/10.1007/s40614-018-0171-8
Kratochwill, T. R., Hitchcock, J., Horner, R. H., Levin, J. R., Odom, S. L., Rindskopf, D. M., & Shadish, W. R. (2010). Single-case designs technical documentation. What Works Clearinghouse. https://ies.ed.gov/ncee/wwc/Docs/ReferenceResources/wwc_scd.pdf
Kroese, D. P., Brereton, T., Taimre, T., & Botev, Z. I. (2014). Why the Monte Carlo method is so important today. Wiley Interdisciplinary Reviews: Computational Statistics, 6(6), 386–392. https://doi.org/10.1002/wics.1314
Kwak, Y. H., & Ingall, L. (2007). Exploring Monte Carlo simulation applications for project management. Risk Management, 9(1), 44–57. https://doi.org/10.1057/palgrave.rm.8250017
Kyonka, E. G. E., Mitchell, S. H., & Bizo, L. A. (2019). Beyond inference by eye: Statistical and graphing practices in JEAB, 1992-2017. Journal of the Experimental Analysis of Behavior, 111(2), 155–165. https://doi.org/10.1002/jeab.509
Lindsley, O. R. (1992). Precision teaching: Discoveries and effects. Journal of Applied Behavior Analysis, 25(1), 51–57. https://doi.org/10.1901/jaba.1992.25-51
Long, C. G., & Hollin, C. R. (1995). Single case design: A critique of methodology and analysis of recent trends. Clinical Psychology & Psychotherapy, 2(3), 177–191. https://doi.org/10.1002/cpp.5640020305
Marchant, N. J., Li, X., & Shaham, Y. (2013). Recent developments in animal models of drug relapse. Current Opinion in Neurobiology, 23(4), 675–683. https://doi.org/10.1016/j.conb.2013.01.003
MathWorks. (2020). MATLAB (Version 9.9). The MathWorks, Inc. https://www.mathworks.com/products/matlab.html
McCullough, B. D., & Heiser, D. A. (2008). On the accuracy of statistical procedures in Microsoft Excel 2007. Computational Statistics & Data Analysis, 52(10), 4570–4578. https://doi.org/10.1016/j.csda.2008.03.004
Mitteer, D. R., Greer, B. D., Randall, K. R., & Briggs, A. M. (2020). Further evaluation of teaching behavior technicians to input data and graph using GraphPad Prism. Behavior Analysis: Research and Practice, 20(2), 81–93. https://doi.org/10.1037/bar0000172
Moeyaert, M., Ugille, M., Ferron, J. M., Beretvas, S. N., & Van den Noortgate, W. (2013). The three-level synthesis of standardized single-subject experimental data: A Monte Carlo simulation study. Multivariate Behavioral Research, 48(5), 719–748. https://doi.org/10.1080/00273171.2013.816621
Odum, A. L., & Shahan, T. A. (2004). d-Amphetamine reinstates behavior previously maintained by food: Importance of context. Behavioural Pharmacology, 15(7), 513–516.
Onghena, P. (2018). Randomization tests or permutation tests? A historical and terminological clarification. In V. Berger (Ed.), Randomization, masking, and allocation concealment (pp. 209–227). Chapman & Hall/CRC Press. https://doi.org/10.1201/9781315305110-14
Peng, C. Y. J., & Chen, L. T. (2018). Handling missing data in single-case studies. Journal of Modern Applied Statistical Methods, 17(1), Article eP2488. https://doi.org/10.22237/jmasm/1525133280
Perone, M. (1991). Experimental design in the analysis of free-operant behavior. In I. H. Iversen & K. A. Lattal (Eds.), Experimental Analysis of Behavior (Part 1, pp. 135–171). Elsevier Science.
Perone, M. (1999). Statistical inference in behavior analysis: Experimental control is better. The Behavior Analyst, 22(2), 109–116. https://doi.org/10.1007/BF03391988
Pustejovsky, J. E. (2018). Using response ratios for meta-analyzing single-case designs with behavioral outcomes. Journal of School Psychology, 68, 99–112. https://doi.org/10.1016/j.jsp.2018.02.003
R Core Team. (2020). R: A language and environment for statistical computing. R Foundation for Statistical Computing https://www.R-project.org/
Shull, R. L. (1999). Statistical inference in behavior analysis: Discussant's remarks. The Behavior Analyst, 22(2), 117–121. https://doi.org/10.1007/BF03391989
Sidman, M. (1960). Tactics of scientific research. Authors Cooperative.
Smith, J. D., Borckardt, J. J., & Nash, M. R. (2012). Inferential precision in single-case time-series data streams: How well does the EM procedure perform when missing observations occur in autocorrelated data? Behavior Therapy, 43(3), 679–685. https://doi.org/10.1016/j.beth.2011.10.001
Trafimow, D. (2019). A frequentist alternative to significance testing, p-values, and confidence intervals. Econometrics, 7(2), 26-40. https://doi.org/10.3390/econometrics7020026
Wasserstein, R. L., Schirm, A. L., & Lazar, N. A. (Eds.) (2019). Special issue on statistical inference in the 21st century: A world beyond p < .05. The American Statistician, 73(1, suppl. 1).
Wickham, H. (2014). Tidy data. Journal of Statistical Software, 59(10), 1-23. https://doi.org/10.18637/jss.v059.i10
Wickham, H. (2016). ggplot2: Elegant graphics for data analysis. Springer-Verlag https://ggplot2.tidyverse.org
Wickham, H., François, R., Henry, L., & Müller, K. (2020). dplyr: A grammar of data manipulation. R package version 1.0.2. https://CRAN.R-project.org/package=dplyr
Young, M. E. (2019). Modern statistical practices in the experimental analysis of behavior: An introduction to the special issue. Journal of the Experimental Analysis of Behavior, 111(2), 149–154. https://doi.org/10.1002/jeab.511
Acknowledgments
The authors thank Kenneth W. Jacobs for productive conversations about Monte Carlo analyses and randomization tests.
Availability of Data and Programming
The app can be found at https://shiny.georgiasouthern.edu/BA_Monte_Carlo/. Data and programming code at the time of publication are archived on the Open Science Framework (https://osf.io/gqtxz/) and the programming code that is running the app will be maintained on GitHub (https://github.com/jefriedel/BA_Monte_Carlo).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Internet
The authors declare no conflict of interest.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Friedel, J.E., Cox, A., Galizio, A. et al. Monte Carlo Analyses for Single-Case Experimental Designs: An Untapped Resource for Applied Behavioral Researchers and Practitioners. Perspect Behav Sci 45, 209–237 (2022). https://doi.org/10.1007/s40614-021-00318-7
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40614-021-00318-7